4/27/2011

Food Traceability Based on Robotic Agriculture- Precision Agriculture Oriented Machines -

Naoshi KondoDivision of Environmental Science and Technology, Graduate School of Agriculture, Kyoto University Kitashirakawa-Oiwakecho, Kyoto 606-8502, Japan

The world is worrying about the quality of food supply.

food poisoning by bacteria, illegal unregistered agricultural chemicals,

camouflage products, BSE, O-157…and Bio-terrorism.

Food Traceability Systemduring production, distribution and consumption

4/27/2011

Robotization in Agriculture

Transplanting robot

Strawberry harvesting robot

Fruit grading robot

Leek pre-processing robot

Robotic sprayer

Real-time Soil SensorAutonomous tractor

Autonomous transplanter

Autonomous combine harvester

Under developmentCommercialized

Field management

Residue process InformationShipping

Seedling production

Plant management

Harvesting

Pre-Processing

Grading

Cutting sticking robotGrafting robot

Measurement BoxPower source, PC,

Spectroscopy,PLC, amp board

Antenna of DGPS

Chisel(sensor probe

housing) ※ Date: 2004. 11. 21 Depth: 150mm, Speed: About 30cm/ secCooperation with SHIBUYA MACHINERY CO.,LTD., TUAT

Commercialized Real time soil sensor

Field management

4/27/2011

Arrangement of Sensors in chisel

Illumination fiber

EC Electrode

Laser Displacement sensor Color camera

Condensed fiberGround surface

Soil flattener

4/27/2011

MoistureContent

NO3-NSOM

pH EC

Real-time soil sensor

N, P, K , EC, pH, moisture content, SOM,soil temperature, compaction

Maps from Soil Sensor

phosphoric acid, potassiumBy courtesy of

Prof. Shibusawa, TUAT

4/27/2011

Autonomous vehicles

Robots keep operation record automatically.

By courtesy of Prof. Noguchi, Hokkaido U.

4/27/2011

Seedling production

operation recordsCrop varietyPlant ID, Color and size of seedlings

Grafting robotCutting sticking robot

Transplanting robot

4/27/2011

Crop management

Spraying robot

Bagging robotCombination of fertilizing robot

and field soil sensor

“information oriented” field

When, where and what kinds of

chemicals, fertilizer???

operation records:irrigation, chemical spray, Fertilization Turf mowing robot

Harvesting robot (strawberry, tomato, cherry tomato, cucumber, eggplant, cabbage, mushroom, orange, grape, hay, rice, wheat and etc.)

4/27/2011

4/27/2011

A strawberry harvesting robot

Robot constitution3 DOF manipulatorSucking and cutting end-effectorStereo vision by use of color CCD camerasDL (Lighting devices) with PL filterRail type traveling device

Product location

Harvested time and date

Crop ID

4/27/2011

A harvesting robot in plant factory

4/27/2011

Other operation robots

Twig cutting robot

Milking robotWool sharing robot

Meat inspection

4/27/2011

Remote cowshed watching service

4/27/2011

PearApplePeachPersimmonTomato

Fruit Grading Robot

4/27/2011

②

Pusher

TV camera

Halfway stage

x

Lifter

①③

④⑤

⑥

⑦

⑧

z

Blower

Fruit providing robot

Gradingrobot

Pusher

Rotation

Constitution of robotic grading system

4/27/2011

Stroke：1165mm

①④

③

⑤

②

100

467

308

290

Path

Speed

① ④③ ⑤② ④

1000mm/s

250 317 308 145 145 (mm)

(s)0.5 0.317 0.683 0.6 0.3 0.3

2.7 s

Cycle time of the stroke：4.25 sItems

2.7 s （Go）

0.4 s （Initial movement）

1.0 s （Return）

0.15 s （wait）

Rotation

Performance of grading robot on a condition of 80% 24 fruit containers and 20% 15fruit containers:12/4.25×3600×0.8=8131 fruits10/4.25×3600×0.2=1694 fruitsTotal 9825fruits/hour/robot

Rotation

Action and performance of fruit grading robot

(mm) Halfway stage

Tray on line

4/27/2011

Information of fruit appearance

Side(1) Side(2)

Side(3) Side(4)

TopBottom

Side(1) Side(2)

Side(3) Side(4)

Top Bottom

Side(1) Side(2)

Side(3) Side(4)

Top Bottom

＋ Internal quality information(Sugar & acid contents, rot, ….etc.)

Original images Color conversion images Processed images

4/27/2011

4/27/2011

0

500

1,000

1,500

2,000

2,500

1 6 11 16 21 26 31 36 41Producer Ｎｏ．

Fruit Ｎｏ．

L1

L2

L3

L4

L5

Grading result of pear on Sep.24, 2002

L�2�

4/27/2011

Machine Vision Technologies in Bioproduction

Oct.21, 2008, AITPL filtering Dome or diffuser

MirrorDiffuserLight

True color HalationWall reflectance

CameraLight

Camera

Reflecting plate

PL filtering Light and Dome

PL filter

4/27/2011

PL Filtering Light

Ready to sell! 10 year warranty“DL” is an SI Seiko-original illumination equipment for image acquisition to make direct lighting possible.

Small power（50W） and high conversion efficiency from electricity into light.

PL filter Heat absorption filters

Fan(more than 1000nm)

(800～1000nm)

Halogen lamp

120 white LEDs

4/27/2011

Direct luminaire (PL) Indirect luminaire (Dome)

True color Inavoidable reflection

4/27/2011 ComparisonTomatoes with plastic package

With drops

Indirect light

With drops

PL

With drops

PL

With drops

Indirect Light

4/27/2011

Outer epidermis cell

Parenchymatous cell(Inner tissue)

Cuticular layer(Wax layer)

Fruit, leaf, stalk

4/27/2011

Popular round fruits

Citrus fruitsTomatoWaxed apple (Tropical fruit)PotatoPersimmonOnionPepper

4/27/2011

Larger round fruits

Water melonMelonPumpkinCabbageLeaf vegetables

4/27/2011

History of the NIR inspection system

In 1989, A NIR sensor to inspect the peach was introduced (Mitsui Co., Ltd.)Yamanashi Pref. (Japan)In 1995, Orange sugar content and acidity

(FANTEC）

At present, NIR sensors more than 300 place grading machines are installedMitsui Co.Ltd. has distributed more than 800 sensors

Price of sensor (1 pc.): $100,000-- 400,000Maker: 3 – 4 companies in Japan

Grading machines which use NIR inspection systems are supported by national government

4/27/2011

Commodity Measurement

Orange Sugar content, Acidity (Void and low moisture content)

Kiwi fruit Sugar content, Acidity (degree of hardness, post harvesting ripening process)

Apple Sugar content, Acidity, Maturity degree, degree of ripeness, rotten portion, (blackheart)

Pear Sugar content, Acidity, Maturity degree, degree of ripeness, rotten portion, (blackheart)

Peach Sugar content, Acidity,

Onion Sugar content, blackheart

Potato Cavity, blet

Water Melon Sugar content ,Ripeness

Melon Sugar content ,Ripeness

Persimmon Sugar content, tanin

Others （Pine apply, Mango, etc)

Measurement details

4/27/2011

System details （Semi-transmissive）

Optical sensor

Lamp

Light transmission carrier

Optical measurement system

Products Apple・Peach・Pear・TomatoWater melon・Melon etc

Gauge control Unit Below orange areaConveyor speed Max 36m～60m/minNumber of Lamps ２～１０

Light source are kept at the side and transmitted light is received at the bottom by optical sensor

4/27/2011

Lamp

System details （Total transmittance）

Products Orange and others

Gauge control Unit Full Internal

Conveyor speed Max 60m/min（5～10／sec）

Number of Lamps １～ onward

In this case fruit is allowed to pass between source and sensor. Transmitted Optical rays are received by the optical sensor

Optical measurement system

Optical sensor

4/27/2011

Other sensors to measure sugar content

Portable type

Handy Type

（Mitsui Co. Ltd）

（FANTEC） （ASTOM）

Can find in the department stores

for consumers’ checking

Use at the farm for producers

4/27/2011

Coffee Break

What is terahertz waves?

Characteristics of THz waves- Permeability - Easy handling- Reasonable spatial resolution - Safe for humans

The spectral region of THz wave between microwave and infrared still remains largely undeveloped mainly because of the lack of proper tunable sources.

4/27/2011

Stop Drug Addiction

Stop Drug Smuggling at

Post Office

By Terahertz (THz)

4/27/2011

Elongated fruits

Eggplant, Cucumber

4/27/2011

Specific longer products

Preprocessing before grading

Leek

4/27/2011

size, color, shape, disease, internal injurysugar and acid contents, residual chemical

4/27/2011

FertilizingChemical sprayingIrrigation……

Accumulation to DB

•Appearance•Internal quality

Data from grading system can be used for precision farming

On Geographical dataGrading systemGrading robot

Farming support system based on GIS

Field information

Grading information

Management information

4/27/2011

Field information

Weather information

Chemical information

Yield information

Quality information

Mapping on GIS dataGeographical information

4/27/2011

Roles of Fruit Grading facility

1. Efficient sorting, and labor saving2. Uniformization of fruit quality3. Enhancing market value of the products

(Establishing local region brand of products)4. Fair payment to producers based

not only on quantity but on quality of each fruit5. Farming guidance from grading results and GIS6. Contribution to the Traceability system

for food safety and security

Producer Operator Distributor Consumer

4/27/2011

Flow of information from field and product

Operation record:Irrigation, fertilization,chemical spraytransplanting, pruning

Plant variety, Crop ID, Size, Color, Disease

3D location of productHarvesting time &date

Crop ID, Size, Color,Disease

Appearance(color, size, shape, bruise, disease)Internal quality (Sugar & acid contents, rotten core, inside defect)Graded rank, Chemical residue, IngredientsDistrict name

Harvesting GradingSeedling productionCrop management

Selling price, selling time, information of arrival of goods, quantity to sell, Opinions of costumers (on taste, freshness….)

Field ID (address, height above sea level), Producer IDClimate, weather information (Temp, humidity, irradiation, precipitation)

Field management

N, P, K, pH, ECSOM, MC, Temp.,Compactness,

Transportation Environmental condition (Temp., humidity)Package type and method, transporting method, time, and distance

Production information

Consumption information

Distribution information

4/27/2011

Traceability System with a fruit grading robot

4/27/2011

ROM-Writer

Main flow on traceability data

TraceabilityDatabase

１

３４

５

７６

２ SizeColorShapeDefectProducer ID

Field IDReceived dateVarietyContainer numberFruit number

PC

Farming operation recordsChemicalsFertilizers……..

Grading robot

Product ID

Sorting

Barcode reader

Packing robot

Product ID

Transportation dataEnvironmental condition

Sales dataPrice, quantity

IC memory

Field Information

Reception ID

Grading ID

Product ID issued date and timePacking robot No.Grading and reception information

http://www.jau-brand.jp

4/27/2011

Soil analysis centerProduct information center

Agricultural Cooperative Association

Soil sensor

Grading robot

Intelligent farmingPrecision farming

Consumer Market (Distribution)

Product informationField information

Farming guidanceDSS for farmers

ResidueCarbonization

BRAND BRANDVoice of consumer Quantity andmarketing value

Analysis of soil and chemicalsGIS, DSS

Variable distribution channelMarketing route

BiomassRe-uses

Transport

Operation recordsSensing information ID tags

FreshProduct

New flow of product and information

Information oriented field Information added product

FPAC, Province

Price Difference between US & Japan

At Meijier, Columbus OH, and Fuji supermarket, Matsuyama, Ehime, June and July, 2005

0100200300400500600700800900

1000

Tomat

o

Cucum

ber

Eggp

lant

Baby

eggp

lant

Color s

w-pe

perLe

ek

Cabba

ge

Lettu

ce

Broc

oryCor

n

Napp

a

Daiko

n

Apple

(Fuji)

Water

melon

Melo

nKiwi

US

JPN

Pric

e (Y

en)

vegetables fruits

4/27/2011

2 dollar melon (Cantaloupe)USA

5 dollar (¥500) melon Japan

Difference between American & Japanese Melons

4/27/2011

More precise in future…

4/27/2011

Mobile fruit grading robot

Field

Canopy

Mobile grading robotN

Course of mobile grading robot

Harvesting date and time, location of product Yield mapQuality mapTree management

Appearance information, Internal quality information

（Information from harvesting robot）

（Information from grading robot）＋

Mobile grading robot for orange fruits

This robot results will be presented soon.

4/27/2011

DL End-effector

ZX

Power unitsServo driversElectric balance

Container

Computer

TV camera

Mobile grading robot for sweet pepper(A 3DOF manipulator, an end-effector,

5 color cameras, 9 Direct Lightsand three photo sensors to give triggers)

4/27/2011

Thank you

Any Questions?

4/27/2011

4/27/2011

Soil Reflectance 400-900 nm for 256 channels900-1770 nm for 128 channels

RTK-GPS Location within 2 cm accuracyMicro CCD Camera Soil Color Image with 512X512 pixLoad Transducer Soil Strength, Cutting ResistanceEC Electrodes Soil Electric Conductivity

Examples of soil images captured every 1m in a paddy field at 30 cm depth.

IR_1525

0.2

0.3

0.4

0.5

0.6

800 1000 1200 1400 1600 1800

Wave length (nm)

Reflecta

nce

系列1

系列2

系列3

系列4

系列5

系列6

系列7

系列8

系列9

系列10

系列11

系列12

系列13

Vis_1525

0

0.1

0.2

0.3

0.4

0.5

400 900

Wave length (nm)

Reflecta

nce

系列1

系列2

系列3

系列4

系列5

系列6

系列7

系列8

系列9

系列10

系列11

系列12

系列13

Examples of soil reflectance collected every 1 m for visible-NIR (left) and NIR (right) wave ranges in an upland field.

Data Collected by the Soil Spectrophotometer

46.8 50.8

48.5 48.6 50.3

51.8 49.0 50.1

53.0 48.4 49.5

51.7 44.5 49.7

49.9 45.0 50.2

49.5 48.0 51.2

50.7 46.3 51.3

51.4 45.8 51.6

49.9 49.5 50.9

50.7 45.9 51.0

48.7 44.6 51.6

48.5

51.4

50.5 47.9

46.2 51.6 48.1

46.4 52.1 48.9

46.9 52.7 50.3

48.3 54.4 50.7

48.9 54.4 51.9

50.5 51.8 52.0

52.3 53.5 53.1

50.0 54.0 54.4

49.2 54.4 54.4

48.2 54.1 56.0

48.1 56.3

47.8 53.7

45.9 43.0 46.3

48.3 47.7 47.4

47.3 50.4 48.7

46.4 52.4 45.9

45.1 49.2 44.9

47.6 47.6 43.6

45.9 45.5 44.1

47.3 44.9 44.3

46.3 44.8 43.7

46.2 43.6 44.2

46.4 44.2 43.4

46.6 44.9 43.0

45.8 45.8 44.5

45.9 46.1 44.4

46.6 47.3 43.8

46.1

44.6 43.6

46.0 44.8 43.9 47.1

45.2 45.1 44.8 42.9

44.0 45.2 46.0 41.5

46.8 45.9 45.2 43.0

47.0 46.4 46.7 46.5

48.4 47.3 47.0 46.8

45.1 48.3 47.6 47.4

47.1 48.5 48.7 47.3

47.2 48.6 47.3 50.3

50.8 48.7 51.4 49.8

53.5 51.1 58.1 51.1

53.3 53.0 55.2 52.8

53.1 55.1 54.9 52.7

55.5 57.9 58.1 56.3

58.1 58.6 60.0

70.0 60.4

40.1 41.1 41.7 43.4 42.3 40.5

40.5 40.7 40.6 39.4 41.1 41.2 40.4

44.1 43.5 43.5 43.7 45.1 44.7

39.4 41.6 42.0 41.6 40.0 40.4 45.4

40.6 41.7 43.0 43.5 42.1 43.9

40.1 40.1 43.4 41.0 40.8 40.2

40.9 40.2 40.1 40.8 41.5 43.1

41.0 40.9 41.7 43.3 45.7 50.6

42.3 41.2 43.0 39.8 44.0 47.6 49.0 45.8 46.9 47.3 45.6 44.8 44.1 44.8 45.2

38.5 40.6 40.7 41.4 42.1 41.2 40.8 40.5 37.8 37.7 38.5 37.7 38.9 40.5

25.8 36.2 40.8 43.1 41.8 40.5 37.6 41.3 39.2 38.4 39.6 39.8 39.3 41.3 44.6

36.2 36.6 38.0 40.5 41.1 38.7 40.5 43.9 41.1 40.7 34.8 43.5 42.7 42.8

49.3 47.0

45.6 44.3 41.9 46.4

46.1 48.3 41.5 47.6

44.5 47.8 40.5 48.5

45.2 47.5 40.6 41.2

45.6 46.3 40.1 42.6

44.1 45.2 41.5 43.9

43.2 46.8 41.3 42.7

43.6 46.8 43.4 43.8

45.0 44.6 43.2 45.6

47.3 50.7 40.9 46.6

43.3

41.5 45.1 44.2 43.8 48.3 45.8 44.2 46.1 47.7 45.4 43.0 44.7

44.0 45.4 46.1 46.5 46.6 46.3 44.9 46.4 46.0 44.6 43.9 44.5

45.3 44.9 45.0 44.8 44.6 45.1 44.5 45.2 44.8 40.9

43.9

40.9 41.7 41.1

40.4 42.6 40.5

42.4 40.2 40.8

41.9 42.1 40.6

41.3 42.8 40.5

39.7 42.7 41.9

39.2 42.6 40.6

39.3 43.2 40.8

39.6 42.5 41.9

38.3 41.2 46.2

34.0 43.9

49.6 62.5

48.2 47.4 53.2 58.2

48.1 47.8 48.9 60.0

46.8 45.0 46.0 57.9

47.2 44.5 46.3 56.9

42.9 48.7 47.7 54.4

43.1 46.0

Unit：μS/ｃｍ

0

135

240

345

450

555

0

20

40

60

80

100

1 2 3 4 5

頻度

　(%

)

0

20

40

60

80

100

1 2 3 4 5

頻度

　(%

)

0

20

40

60

80

100

1 2 3 4 5

頻度

　(%

)

0

20

40

60

80

100

1 2 3 4 5

頻度

　(%

)

0

20

40

60

80

100

1 2 3 4 5

頻度

　(%

)

0

20

40

60

80

100

1 2 3 4 5

頻度

　(%

)

0

20

40

60

80

100

1 2 3 4 5

頻度

　(%

)

0

20

40

60

80

100

1 2 3 4 5

頻度

　(%

)

0

20

40

60

80

100

1 2 3 4 5

頻度

　(%

)

0

20

40

60

80

100

1 2 3 4 5

頻度

　(%

)

0

20

40

60

80

100

1 2 3 4 5

頻度

　(%

)

148149 91131

144

147

157

120-1 120-2

50

49

Average： 49.3

Average： 51.4

Average： 46.0

Average： 49.3

Average： 41.9 Average： 42.0

Average： 44.7

Average： 41.1

Average： 45.1 Average： 41.5 Average： 49.9

Soil Variability Maps : MC, pH, EC, TC, TN

4/27/2011

4/27/2011

Display during measurement

Visible regionＮＩＲ

Subsurface Soil image

Spectral Data

START

STOP

FINISH

Compactness

Fluctuation on surface

Latitude

Longitude

Image Analog DataSpectrumMeasureStop