vlbi session pre-checks for mark iv, vlba and dbe systems · readings of 0.1–2.0 on the...

[Type here] 1 [Type here]

VLBI Session Pre-Checks for Mark IV, VLBA and DBE Systems

See also: “Basic Checklists” section of Mark IV Field System manual

Sample checklist in Operations workshop notes

Draft VGOS set-up in Operations workshop notes

Please note that all references to Mark IV apply to Mark III as well unless specific separate

instructions are given for Mark III. The same applies to VLBA and VLBA4.

Experiment SNAP and procedure files

• Run DRUDG on the session (.skd or .drg) file to create the SNAP and procedure file.

Print hardcopies of the schedule and session notes.

• Edit the session procedure file to insert station-specific parameters such as IF

distributor attenuator settings.

• Be sure to create a new procedure file for each session from the .skd or .drg file. Do

not rely on default set-up procedures in the station procedure file in case system set-up

parameters have changed such as the frequency sequence.

Computer control

• Check that the FS is controlling the VLBI terminal and antenna.

o On a Mark IV terminal, check that the remote/local switches on all the

modules are set to remote.

• Set up the VLBI terminal with the session set-up procedure (e.g., setupa).

• Refer to the FS VGOS Operations notes, which follow the plots at the end of this

section..

Station timing

• Check that the formatter time agrees with GPS time at the one-second level. If it does

not, use ‘fmset’ to adjust the formatter time. For Mark III you must set the time

manually.

• If NTP is reliably available, the FS computer should run on NTP (with both local and

remote stratum one servers). The FS time.ctl control file should be set to use the

"computer" model. Experiment operations should not begin after a computer reboot

until NTP is sync'd. Use the check_ntp SNAP procedure to verify that NTP is

sync’d; one server must have an asterisk ‘*’ in front of it. Please see notes for the TOW

lecture Impact of Operations on Data Analysis for details about set-up and use of

check_ntp.


• If NTP is not reliably available, check that the FS clock is correct to <1 second. If it is

not, enter the command sy=run setcl offset &. This command, should also

be issued, whenever the formatter time is reset. If ‘fmset’ is used to set the formatter

time, the FS time, will automatically be reset when ‘fmset’ exits.

• If a computer other than the FS computer controls the antenna pointing, check that its

timing agrees with GPS to <1 second.

• For RDBEs, to check time and offsets use time SNAP procedure. Run 'fmset' if

necessary. Cycle through bands a,b,c,d (Avoid "s"ync) and exit.

• Using a counter with time interval measurement accuracy <100 ns, measure the time

interval between the 1pps signals from

o formatter clock and GPS receiver

o maser and GPS receiver.

The formatter should be within a few microseconds of GPS.

• Check that the formatter/GPS/maser 1pps offsets are consistent with other recent

measurements.

Feed polarization

• Ensure that the feed polarization is correct.

• For geodetic S/X observations, the correct polarization is RCP.

• For VGOS use linear X and Y.

Receiver

• Check that the physical temperatures of the 20 K and 70 K stations in the Dewar are

within their normal ranges and are not rising.

• Check that other receiver parameters that can be monitored, such as FET currents, LO

and phase cal temperatures, power levels, power supply voltages, etc., are within their

normal ranges.

• Check that the receiver LO is set to the correct frequency.

• If there is any uncertainty about the LO frequency, measure it.

o If the LO is accessible, measure its frequency directly with a counter.

o Alternatively, radiate a coherent test tone at a frequency in the RF passband

toward the antenna and measure its frequency in the IF or at baseband. Suitable

test tones can be generated up to a few GHz by using an amplifier and frequency

multiplier on the LO monitor output of a BBC or VC.


IF distributor/IF3 module/IF upconverter or downconverter

• Check that the inputs to these modules are connected to the correct IF signals.

• In a Mark IV DAT, check that the VC inputs are connected to the correct IF signals

(IF1/IF2 low/high or IF3). The IF switch box controlled through the IF3 module can be

used to switch the VC03 and VC10 inputs automatically between low and high bands.

• For RDBEs all 'sigma' values should be close to 20.

Video or baseband converters

• Measure the frequency of each VC/BBC LO with a frequency counter and compare

against the correct value.

• Check that the baseband bandwidth in each VC/BBC is correct.

• In a VLBA DAR, check that the IF input to each BBC is correct.

• Check that the baseband power levels are in the proper range.

o In a Mark IV DAT, adjust the IFD/IF3 attenuation to give USB and LSB power

readings of 0.1–2.0 on the front-panel meter of each VC, with the 10-dB

attenuators in. Use the largest amount of IFD/IF3 attenuation that keeps all

channels being recorded above 0.1 volt.

▪ If readings of 0.1–2.0 cannot be achieved in all VCs, as a temporary

measure the lower limit can be relaxed to 0.05, or small (1–3 dB) SMA

attenuators can be added to the inputs to the stronger VCs.

▪ Do not use large (≥10 dB) attenuators on the VC inputs, as those VCs

without attenuators are liable to saturate.

o In a VLBA DAR, check that the baseband power readings are all approximately

16000 counts. The AGC gain in each BBC should be between –10 dB

and +10 dB.

o The proper long-term solution to strongly frequency-dependent VC power

readings or BBC gains is to flatten the IF passband with appropriate filtering.

Please find this describe in more detail in the Ops Impact on Analysis chapter.

• For a Mark IV system, the command ‘ifadjust’ may be used to set the attenuators

automatically, provided the VC TPI readings are working and RFI is not severe. The

DAT must first be configured with the set-up procedure from the experiment procedure

library. The antenna should be pointed near zenith (or toward any direction that gives

nearly the minimum system temperatures), and the weather should be clear. If ‘ifadjust’

converges, the results should be the optimum attenuation for that mode. (No changes

should be necessary during the experiment as the source elevation or the weather

changes.) The resulting values should be edited into the IFD set-up procedure for the

mode. If there is more than one mode in an experiment, this procedure should be

repeated for each mode.

• For VGOS RDBE use FS 'setupbb' and 'auto'.


Antenna focus

• For geodetic and astrometric observations, lock the focus at the position that gives

peak response at mid-elevations.

• For astronomy-only (i.e., source mapping) observations, the focus may be adjusted

with elevation angle to give maximum gain.

Antenna pointing

• Check antenna pointing on two or more widely-separated sources.

• Pointing errors in both axes should be smaller than beamwidth/10 at the highest

observation frequency.

• The FS command ‘fivept’ may be used to automate the measurement.

System Equivalent Flux Density (SEFD)

• Measure the SEFD in each frequency band and compare against the nominal value.

o Using a strong radio source of known flux density, measure Tsource

/Tsys

.

o Convert to SEFD by dividing Tsource

/Tsys

into the source flux density

(corrected for size effect if the source angular extent is comparable to, or larger

than, the antenna beamwidth).

o The FS command ‘onoff’ may be used to automate the measurement.

System temperatures

• With the antenna pointed near zenith, measure the system temperatures in all

VC/BBC and broadband IF channels, and compare against nominal values.

Phase calibration signal

• Use an oscilloscope and 10–kHz viewing filter to examine the baseband phase cal

signal in each USB channel. (This section assumes that phase cal is present at 10 kHz

in each baseband USB channel, as will be the case with a 1 MHz phase cal pulse rate

and a total LO frequency of nnn.99 MHz.) Trigger the oscilloscope with a timing signal

derived from the maser (e.g., the frame sync VS-14 module).

o Phase cal should be present with the proper amplitude.

o Phase cal should be stable in phase relative to the maser timing signal.

• Use the same equipment to examine each LSB channel.

o The LSB phase cal level should be >20 dB weaker than in USB.


• A good end-to-end system test is to check for phase cal in the bit stream from each

USB data track.

o For a Mark IV system, connect one of the decoder DATA outputs to the 10–

kHz viewing filter input. Enter the commands ‘enable=s1’ and ‘st=for,0,on’.

Select USB tracks with the ‘repro=byp,<trackA>,<trackB>,,<rate>’ command

and check for phase cal on the oscilloscope.

o For a VLBA system, use the ‘repro’ and ‘dqa’ commands to measure the

phase

cal amplitude in each USB channel.

o For VGOS RDBE run RDBE monitor program.

Cable calibration

• The cable counter should be operating in single–sample mode, not averaging.

• The cable counter reading should be in the normal range.

• Single–sample readings should be stable to ±1 μsec.

• Check that the cable calibration system measures cable length changes correctly.

o Log the cable reading.

o Insert a short cable length in the cable from the ground unit to the antenna

unit, and log the cable reading again.

o Remove the short cable and log the cable reading again.

o Compare the change in readings with the nominal value.

Mark 5A

Please note that Mark-5A operational procedures are in the TOW07 notebook. Mark-6 will be in the

TOW15 notebook. Additional guidelines for both the Mark-5 and Mark-6 can be found in the Systems

failure chapter of the TOW15 notebook.

• 8-Pack Installation –

1. Remove the 8-pack from the shipping box and their shipping covers.

2. Inspect for damage.

3. Pull front lever down so that it is horizontal.

4. Slide 8-pack easily into Mark5A machine until it stops.

5. Lift front lever which will insert the 8-pack into the connector.

6. Turn the key switch to the right until it stops.

7. Repeat with second 8-pack.

• Conditioning the 8-Pack –

If there is time each 8-pack that is received at a station should be conditioned before it


is used to verify there has been no shipping damage. This can be done in any appropriate

length gap in recording (4 hours or more) or left running overnight or over a weekend.

Please see ftp://web.haystack.mit.edu/pub/mark5/Conditioning.txt for more

information. A copy of this document follows in this section. Please note that the

SSErase program is run outside the Mark 5A. If the 8-pack is write protected, you will

be prompted to remove the write protection, and you should do so. When the

conditioning is complete, please check the results to confirm that the 8-pack is okay

before using it.

• Mark5A Start-Up –

1. From the Mark5A terminal, login as oper.

2. Enter the password.

3. From the prompt, start a script running by entering ‘script -f anyname’.

4. To start the Mark5A running, enter ‘Mark5A -m0 -f0 &’.

5. To stop the Mark5A, enter ‘EndM5’.

• Field System Checks –

1. Test the communications with the Mark5A by entering ‘disk_serial’ at the Field

System prompt. Sample output:

/disc_serial/VNVF11G6G9BS9L,VNVF11G6G9BW6L,VNVF11G6G99V7L,VNVF11G6G9BVDL,VNVF11G6G9BUVL,VNVF11G6G9A2HL,NVF11G6G99XZL,VNVF11G6G9A2XL,

2. Enter ‘disk_pos’. This will give the position of the disk pointers. Sample

output: 2003.245.16:42:53.42/disc_pos/81952,0,

3. Enter ‘mk5=VSN?’. This will check if a VSN is assigned to the 8-pack.

4. Erase the disk with ‘mk5=reset=erase’. Be very careful that the disk installed

should be erased before entering this command.

5. Enter ‘disk_pos’. This will respond with the disk pointers at 81952,0 (or it may

be 0,0, depending on the version of erase).

6. Set up the procedure file that will be used for the session, e.g., ‘proc=r1087wf’.

7. Issue the set-up command from the session, e.g., ‘setup4f’.

8. Enter ‘disk_record=on’.

9. Enter ‘disk_pos’. Repeat this command many times to confirm recording.

10. Enter ‘disk_record=off’. This stops the recording.

11. Enter ‘scan_check’. Sample output:

/scan_check/2,r1172_0552+398_115-1741,mark4,32,2005y115d17h41m1.080s,49.1s,8.000000,0

12. Erase the disk with ‘mk5=reset=erase’.


Meteorological sensors

• Check that the values of barometric pressure, temperature, and relative humidity

reported by the ‘wx’ command are reasonable and repeatable.

‘Ready’ message

• Approximately an hour before the session is to start, or as soon thereafter as possible,

send a “ready” email message containing the following information:

o comments on any unusal conditions

o pointing offsets and source az/el

o SEFDs and source az/el

o system temperature for each IF

o formatter offset from GPS

o source and epoch of first scheduled observation

o weather and sky conditions

See the Operations workshop notes for more information.

More extensive testing

• After any equipment changes or if a long period of time has elapsed since the last VLBI

session, more extensive testing of the VLBI systems should be carried out. More

rigorous testing should also be conducted on a periodic basis (monthly to annually).

Such tests include:

o Measure pointing offsets over the full sky.

o Check for RFI and for jumps in the phase cal phase and cable cal as the

antenna is slewed across the sky.

o Calibrate the meteorological sensors, especially the barometer.

o Measure the DC level and AC ripple in the DAT/DAR power supplies.

o Measure the strength of the phase cal signal by observing how much the IF or

baseband power level changes when phase cal is turned off.

o Measure the levels of spurious phase cal signals:

▪ with phase cal turned off via the ground unit switch

▪ with phase cal turned on and the receiver LO unlocked

▪ with the cable driving the antenna unit disconnected from the ground

unit.


o For each VC/BBC:

▪ Check that the LO will lock over the full design frequency range (100–

500 MHz for a VC, 500–1000 MHz for a BBC).

▪ Measure the LO phase noise at 2-3 frequencies spanning the design

range.

▪ Measure the USB/LSB image rejection at the upper and lower ends and

middle of the baseband frequency range.

▪ Check each sideband signal on a spectrum analyzer for anomalous shape.

• Monthly execution of the ‘overnite’ procedure is recommended to check the stability of the

VLBI systems. The procedure should be run for at least 12 hours, and preferably 24 hours,

with the antenna stationary. At a minimum, the values of the following quantities should

logged at least every 15 minutes:

o wx

o cable

o formatter-GPS time interval

o system temperatures in all channels

o phase cal amplitude and phase in at least 1 channel per frequency band

o receiver parameters: dewar temperatures, phase cal temp, etc.

o other station-specific parameters


IVS Operations workshop TOW 2017

1.0 Starting a schedule

The basic steps in starting a schedule are:

1.1. Reporting problems when you need urgent help.

An e-mail list has been set up for items needing urgent attention by the IVS Coordinating

Center. The address is: [email protected]

This list should be used by anyone needing urgent attention by or advice from the

Coordinating Center. You can expect an answer normally within a few hours, if the

message is sent during US East Coast working hours. Messages sent on weekends or after

hours may take a little longer.

Please use this address rather than sending messages to ivs-ops or ivs-stations. If the reply

from the Coordinating Center is of interest to a broader audience, we will copy the

appropriate lists on the response.

Anyone can post a message to this list. The only recipients for this list are the Coordinating

Center staff: Dirk Behrend, Ed Himwich, Cynthia Thomas, Mario Bérubé, and John

Gipson.

Examples of the types of messages that should be sent to this list are: problems with

schedules, problems running DRUDG or the Field System, schedules that seem to be made

for the wrong date, schedules that do not include the right stations, operational problems

during observing, and any urgent issue that impacts operations and you need advice on how

to handle.

1.2. Review session notes

This provides one last opportunity to verify that nothing special has been missed in the set-

up for this experiment.

1.3. Correct cable wrap for AZ/EL antennas

For AZ/EL antennas it is important to verify that the antenna is on the correct cable wrap

for the first source. If the antenna starts on the wrong wrap it is all but guaranteed that at

some point in the schedule one or more scans will be lost because the antenna will have to

unwrap unexpectedly.

The wrap for each source is shown in the listings and in the "source=…" commands for

each source. You can perhaps get the antenna onto the right wrap by either driving it

manually or giving it a series of source commands that will "walk it around." Please be sure

to look into this issue far enough before when you start the schedule so that

mailto:[email protected]


you will have time to swing the antenna around if it is needed.

1.4. Make sure there is no old test log on disk

If you made a test running of the schedule be sure to delete the test log file before starting

the schedule for real.

If Mark5B is being used, load both decks with an empty eight-pack and verify the Mark-5B

system is operational. The Mark-5B verification and use notes can be found in this handbook.

The Mark-6 check-out can be found in the VGOS Draft at the end of this chapter.

1.6. Start schedule with "schedule=…"

The preferred form of the "schedule" command uses the line number to start with, e.g.,

"schedule=na225gc,#1", to start at line 1. Start the schedule as early in the

pre-session checklist as possible. This will allow you to record as many of the checks as

possible, directly in the log, with comments.

1.7. Check cable sign. The key points are:

1.7.1. Do the cable check before or after the experiment in the experiment log. Do not do

it during the experiment as this is likely to cause problems.

1.7.2. Check that the cable counter is not using averaging

Averaging should not be needed. The cable counter should be stable at the 1 μsecond

(about 1 mm of one-way cable delay) level without averaging; which is more than

sufficient. If it isn't stable at this level there is probably some problem that needs to

be corrected. Using averaging would only obscure a problem that needs to be

corrected instead of providing any useful improved accuracy.

1.7.3. Do a "cable" with the normal cable.

This establishes the "normal" reading.

1.7.4. Install a short calibrated extension cable

You should have a short cable with a male N connector on one end and a female N

connector on the other end. A cable of 30 cm will give a reading change of about 600

microseconds.

1.7.5. Do a "cablelong" command once the cable calibrator has stabilized again


This establishes the "long" reading.

1.7.6 Do a cablediff command.

This will automatically find the difference and report the sign. If the difference was

as expected, enter a comment to that effect. A suggested format for the comment is

"change of xxx usecs is nominal, longer cable increases/decreases reading".

If the difference was not as expected, you should troubleshoot the cable cal system if

you have time.

1.7.7. Remove the extension cable

Return to normal cable length. If you leave the extension cable in by mistake, please

do not remove it after the experiment starts. Leaving it in will cause no harm, taking it

out is likely too.

1.7.8. Do a "cable" to make sure everything is back to normal

This is final check that everything is back to normal. You should get approximately

the same reading as before.

1.8. Finish start check-list

Complete any remaining items in the pre-experiment checklist. A copy of a sample

checklist is included at the end of this workshop write-up as a reference.

1.9 Send the Ready message. The FS utility msg will send the information in a standard format

and is recommended.

This message summaries your status and lets the world know you are ready. It should be

sent to [email protected]. Be sure to include the session name in the subject line.

Please include the following information.

1.9.1 Comments about anything that may affect the data quality. If you have more than

one polarization available it is recommended that you check the polarization to make

sure it is correct and note that it has be checked.

1.9.2 The source and start time for the first scan. This assists the coordinator center in

determining if you have the right schedule if an update was issued.

1.9.3 Formatter to GPS clock offset. This is needed by the correlator to find fringes. If the

Formatter 1 PPS starts the measurement please report this as the Formatter to GPS

offset; if GPS 1 PPS starts, please report it as the GPS to Formatter offset.

1.9.4 WX: temperature, pressure, humidity, and sky conditions. This assists the analysts in

understanding the atmospheric conditions and interpreting the SEFD and Tsys data.



1.9.5 Cable difference. Please report whether a longer cable makes the reading larger or

smaller. Please also report the difference you observed in the reading when you lengthened the cable and whether it was nominal (as expected) or not.

1.9.6 Pointing values: Please report both X and S SEFDs, source used for measurement, its

azimuth and elevation when measured, and the pointing offsets on each axis. You

can include as many sources as you think are useful.

1.9.7 Tsys values. Please report the system temperature for each IF: X (lower X), S and X2

(upper X). If you have interference you may need to report the values from a video

(base-band) channel for the effected IFs.

You can refer to the Summary of Ops Messages near the end of the notes for the Operations

workshop. That summary includes an example of the recommended format for Pointing and

Tsys data.

If your FS PC has sendmail installed you can use the graphical FS utility msg to send

ready, Start, and Stop messages. It can be configured to add all the information to the

experiment log. Please note there are reports of problems with msg utility if you use the

“keypad” (typically on the right side of keyboards, sometimes used for numeric entry). This

appears to be due to a bug in the operating system or its utilities.

1.10. Send the Start message

This lets the world know that you started and whether there are any problems. Any

additional useful information for the record doesn't hurt either. The message should be sent

to [email protected]. Be sure to include the session name in the subject line.

1.10.1. When started and first source

1.10.2. Any problems

2.0 Making comments in the log Making comments in the log when problems occur is one of the most important functions

of an operator. It is the primary way of information about problems to the correlators (for

some correlators, in particular the VLBA, it is the only way). Please provide as full a

description of a problem and its consequences as possible. Many of the data analysts are not

familiar with the acquisition hardware. It is important to describe the implications in high

level terms, such as which channels are affected. For example, don't just say "A and B cable

switched", say "A and B cables switched; cables were wrong from the beginning of the

experiment until now; the IF channels had the wrong signals; X and S band were reversed;

no valid data in any channel until now.”



Entering comments as a problem is being trouble-shot is desirable since it creates a record

of what happened. Once a problem is resolved or by the end of the experiment in any event,

please make sure the following information has been entered in comments:

2.1. The nature of the problem

2.2. How it was resolved if it was

2.3. The consequences of the problem: how the data was affected.

2.4. The time period for which data was affected.

2.5. TOW Failure Recover class will cover the most common failures.

3.0 Periodic monitoring I: each scan correct:

For each scan the following information should be verified:

3.1. source

3.2. mode (configuration), frequency sequence, formatter mode, IF patching

3.3. on-source

4.0 Periodic monitoring II: hourly (at least)

These items need to be checked less frequently than every scan. Every hour is a reasonable

interval.

4.1. Sky conditions entered as comments in log Cloud cover, wind, precipitation, etc.

4.2. Phase-cal present in all VCs

Visual checks of the phase-cal with 10 kHz viewing filter is desirable. Displaying the

values from a phase-cal extraction system is helpful if they are available as well. The phase

should be stable and the amplitude at the expected level. An occasional check of the

sidebands that are not supposed to have a signal is desirable to verify that the image

rejection mixer hasn't failed. Use the phase-cal check for VGOS.

4.3. Formatter time/date correct

A comparison of the formatter to GPS is desirable. If the formatter doesn't have a display, it

is recommended that you NOT run "fmset" to do this check. Instead check that the error

reported by "setcl" (last entry of the first line of output) is small. And compare the FS time

displayed by "monit2" (System Status Display) agrees with GPS. You should also check

the pointing computer's time it if has one and is used for pointing calculations. The "setcl"

error output is useful for tracking jumps of integer seconds in the formatter time.


4.4. Formatter-GPS offset reasonable

The Formatter-GPS offset should not show any significant jumps in value. If you are

monitoring other clock offsets, e.g., Formatter-Maser, Maser-GPS, or Maser-Other clock,

these should be checked too. It is desirable to have at least one loop, e.g., Formatter-GPS,

GPS-Maser, and Maser-Formatter, to help isolate the guilty party if a jump is detected.

Since the clock offsets are based on 1 PPS signals they are useful for detecting jumps of

less than one second.

4.5. Cable and WX readings reasonable

These readings should be checked to make sure the displayed values haven't jumped outside

a reasonable range.

4.6. Tsys reasonable in all VCs

Monitored Tsys values give some indication of whether the system temperature is

remaining stable and hence whether a problem with the receiver may be developing. One

way to think about these values is that they provide a less complete, but still useful, check

on the sensitivity of the system, when a full SEFD measurement is not possible, such as

during observations in a schedule. The interpretation of the Tsys data requires a little care

since it may become elevated because of weather conditions (particularly at X-Band) or

because of ground pick-up on low elevation scans.

4.7. Disk pack full?

Check get_stats? output to see whether a disk pack has filled up and if so, swap it out with a

fresh one. The disk pack that is not being recorded should always be a fresh one and a full

disk pack should always be swapped out as soon as possible. This applies even if the one

currently being recorded “should” be the last.


5.0 Each shift change III: (at least)

This section describes some things that should be monitored even less frequently than once

an hour. Mostly this is because it is necessary to develop some history of data to look for

trends and outliers. It is useful to run "logpl" to plot these values. With a command file,

"logpl" can generate all the plots automatically for you to view when they are ready. See

the section on "logpl" for more information on running it.

5.1. WX parameters

5.2. Cable

5.3. Tsys, all IFs

5.4. Clock offsets

5.5. Other station specific values

5.6. For disk, check remaining disk capacity

6.0 Rack electronic failures

This may include failure of power supplies, BCC OR VC modules, formatters and Dewars,

disc packs and more. All impact data collection and analysis. The decision to continue

observing or stop and repair the system depends on the nature of the failure. Dewar failures

and Base Band or Video converter problems cause data loss and is described in detail in the

Ops Impact on Analysis chapter.

7.0 Clock Jumps

Handling clock jumps is a somewhat delicate operation. The handling depends on which

clock jumped. If the formatter jumped, but appears now to be stable, there is normally no

point in resetting the formatter since this will just introduce another clock jump. Except in

specific circumstances, described in the last paragraph of this section labeled as LARGE

FORMATTER CLOCK JUMPS, the formatter should not be reset. If the Maser or GPS

or some other clock jumps, it is even more obvious that it is probably best to not make any

changes to the formatter time. In any event the jump, which clock is affected, and the span

of data affected by the jump, should be noted in comments in the log.

It is useful, mandatory really, for identifying clock jumps to be comparing at least three

different clocks, e.g., formatter, Maser, and GPS. It is unlikely that two will jump at the same

time. Of course monitoring even more clocks will reduce the probability further of being

fooled by such an event.

To remove the error message generated by setcl for a clock that doesn’t need to be

corrected, please refer to the /usr2/fs/misc/fstime.txt file. The section on HANDLING


PROBLEMS and SETCL (the case for “a positive or negative integer or zero”) are useful in

this regard. This document contains other useful information about how time is used in the

FS and is recommended reading.

LARGE FORMATTER CLOCK JUMPS: If the formatter has jumped and the new

offset results in the sub-second portion of the Formatter to GPS offset (as reported by gps-

fmout or fmout-gps) greater than ±500 milliseconds or the integer second portion (as

reported by sy=run setcl &) is greater than ±5 seconds, the formatter clock should be

reset. A sub-second offset of greater than ±500 milliseconds will cause confusion about the

actual second. Integer second offsets greater than ±5 seconds may cause loss of SNR.

However, if the jump is not as large as described in this paragraph, the formatter should not

be reset, but monitored closely instead.

8.0 Power Failure recovery

Power failure (or general disaster) recovery can only be learned by practicing it and this

highly recommended. A detailed discussion of the basics for doing it is included in the

Experiment Operations manual in Vol. 1 of the FS Manuals. Some key concepts are

presented here.

8.1. Determine the next source

The most important issue is to pick a source that will give you enough time to get

everything set-up, and the antenna positioned. You should probably get all the equipment

up and working before trying to pick the next source, unless your antenna moves very

"majestically", i.e., slowly. In that case the time required to move the antenna may

dominate everything else. Consider the following factors when selecting the next source to

observe:

8.1.1. Where is the antenna?

Consider how long will it take the antenna to slew from its current position to the new

source. If you have an AZ/EL antenna be sure to make sure that you have considered

whether you will need to force the antenna to a different cable wrap.

8.1.2. Mark 5A Setup

Mark5A presents some special considerations after a power failure. The following

guidelines and Mark5A commands will help save pre-recorded data and allow recording the

session to continue after restarting the mark5A program.

If recording is aborted because of some problem before the "disk_record=off" command is

executed by the Mark 5A, then all data recorded since the previous "disk_record=on" may

be lost. If you are using the "2004y351d" (16December2004) version, or a later version, of

the Mark5A program, then as described below, in most of these cases, it is possible to

update the record pointer (but not the directory) on the disk pack to reflect the data that had

been recorded. This procedure will prevent the data from being overwritten by new

recordings and allow the data from the aborted recording to be correlated. This command


seems to work well in at least two cases: (1) the Mark5A program is ended prematurely by

crash or some other problem or (2) the power is lost to the Mark5A unit. It does not seem

work as well in the case where the key switch is turned off during recording (care should be

exercised of course to avoid turning the key switch on the selected disk pack).

Regardless of the cause of the aborted recording, the best recovery procedure is to execute

the following FS commands (after re-starting the Mark5A program):

mk5=bank_set=<bank with aborted recording>;

mk5=recover=0;

If you do not know which bank has the aborted recording, it is probably the one that is

neither empty nor full. You can use the monit5 'disk space remaining' display and the

'bank_status' command to see which bank is neither empty (100% remaining) nor full

(0.x% remaining). If recording aborted during the first scan, then that bank would show

100% remaining.

It is probably not worthwhile to recover short recordings particularly if doing so would

prevent other scans from being recorded. However, for longer recordings, particularly if

several scans have been recorded "continuously" and a problem prevented a proper

"disk_record=off", recovery may be helpful. Since at this time directory information is not

recovered properly, this won't be helpful for scans transferred off-line after an experiment

with software that depends on the directory information. In that case, the transfer will not

work, but this will at least preserve the data on disk.

The Mark5A "recover" command is available with the "2004y351d" (16December2004) or

later versions of the Mark5A program. It is documented in the "Mark 5A command set"

document:

http://www.haystack.mit.edu/tech/vlbi/mark5/docs/command5a.pdf

8.2. Position antenna.

Once it has been determined which source to use, position the antenna in preparation for

restarting.

8.3. Restart schedule with “schedule=...,#n,1” line number form

The value of "n" should be the starting line from the listing of the intended next

observation. The effect of using "1" as the number of lines to execute will be to only

command the source. While the antenna is moving, you can use the "list" command to see

if there are any commands coming that you want to avoid. If you find there are no

commands you want to skip, you can restart the schedule with a "cont" command. If you do

want to skip some commands, then restart the schedule at the line number where you want

to restart.

http://www.haystack.mit.edu/tech/vlbi/mark5/docs/command5a.pdf


9.0 Post experiment

The post-experiment checks provide an opportunity to "close-the-loop" on the

pre-experiment checks and verify that some things that cannot be checked during the

experiment, such as the SEFDs, are still okay.

9.1. Check cable sign again

This is repeated for good measure in case either there was some problem with the initial

check. This also verifies that the cable system still detects the calibrated cable correctly and

hence has not failed during the experiment. Please see the "Start schedule" section for more

details on how to do the check.

9.2. Complete end check-list

There are several other items on the end of experiment check-list. This should all be

completed. The results of some of the tests will go in the End Message. A sample post-

experiment checklist is included at the end of this write-up for reference.

9.3. Send the Stop Message

The End Message lets the world know you have completed the experiment and your status

at the end. This is especially useful for the correlator since it will help them prepare for

correlating the data. This information should all be reported in comments in the log as well.

It should be sent to [email protected]. Be sure to include the session name in the

subject line. The End Message should have the following components:

9.3.1. Scans missed

List the scans missed, not by line numbers, but by the time spans that data was lost for.

9.3.2. Comments

Please describe any unusual conditions that affected data collection. Please describe any

equipment problems. This might not be limited to things that affected the data collection,

but also include items that require maintenance.

Please include any other information that you think might be useful for the correlator or the

data analysts.

9.3.3. Stop time

Indicate the time data collection stopped.



9.3.4. Clock offset information

Report the final clock offset information. Use the same format as described above for the

Ready message. Reporting the clock offset a second time may seem redundant but it has

two purposes. First in case there was an error in reporting the first value, it provides a back-

up. Secondly if there was no error in the first value it can be used to a rough estimate of the

clock drift rate. Please also include any information about clock jumps and the time span of

data affected by any jumps.

9.3.5. 8-pack inventory

Report the module inventory in the ‘End” message.

9.4. Close log

As with the pre-experiment tests, the post-experiment tests should be done as much as

possible in the experiment log. Once the post-experiment tests have been completed, you

can close the log, either using the "schedule=…" command with a null or

non-existent schedule, by changing the log file with a "log=station" command, or by

terminating the FS.

9.5. Delog parameters for the record

Use "logpl" to delog the ancillary data, cable, WX, clock offsets, comments, etc. It is best to

do this before sending the end message so that any problems that are discovered from the

plots can be reported in the End Message. If you do discover problems, open the log again

with the "log=…" command and add explanatory comments. DO NOT EDIT THE LOG

MANUALLY. (The log is intended to be a record of what happened during the experiment.

If you edit it, and change or delete anything, you defeat this purpose. No one minds if there

are some start-up or other errors in the log. It is more important for the log to be a faithful

record of what happened than for it to be neat.) Save the hardcopy plots along with the

listings you printed which might include handwritten notes. See the section below about

running "logpl".

9.6. Transfer log to BKG/OPAR/CDDIS/aspen

The log needs to be placed on the appropriate server depending on which correlator will

handle the data.

9.7. Send 8-pack(s) to correlator

Prepare the 8-pack(s) for shipping.


9.8 Enter Mark 5 module shipping information into the TRACK program. A new program,

PACKTRACK is currently being developed to also handle Mark 6 modules.

9.9. Send the Wrap-up Message

This message contains the last few items related to an experiment. You can optionally include

this information in the Stop message. It should be sent to

[email protected]. Be sure to include the session name in the subject line.

9.9.1. Log Placement Info

Report where the log was placed: server, directory, and file name.

9.9.2. Additional Shipping Information

If you have any additional information about the shipping such as the routing, this would be a

good place to send it.

9.9.3 Additional Experiments Comments

If you have become aware of any additional problems or if there is any special information you

either have discovered since the experiment ended or forgot to place in the Stop message, this

is a good place to send it. You can of course send follow-up information as often as necessary.



10.0 Sample Checklist

Station experiment check list, Experiment Date

START

1. Pointing: Sky Conditions

Source Az/El Offsets SEFDs: X/S

/ / /

/ / /

/ / /

2. TSYS IF 1(A) IF 2(B) IF 3(C)

3. Cable sign with comments in the log

normal

long

difference, nominal , okay?

longer cable: increases decreases reading

extra cable removed?

4. WX temperature Pressure Humidity

5. Formatter time

6. formatter-GPS offset Counter Starts: GPS FMOUT

7. MASER -GPS offset Counter Starts: GPS MASER

8. VCs phase-cal Levels Frequencies

9. Local checks

10. Send "ready" e-mail with TSYSs, SEFDs, pointing offsets, source, az-el, weather, sky

conditions, gps-fmout offset, first source and time, recorder humidity, any problems,

confirmation of any unusual set-up, equipment condition

11. send "start" message

EVERY HOUR or more often

1. enter WX and sky condition comments

2. Check Phase-cal all VCs

3. Check VC signal levels

4. Check system temperatures

5. check formatter time

6. formatter offset reasonable

7. Check cable value reasonable

8. Check WX reasonable

9. Parity errors okay

END

1. Pointing: Sky Conditions

Source Az/El Offsets SEFDs: X/S

/ / /

2. TSYS IF 1(A) IF 2(B) IF 3(C)

3. Cable sign with comments in the log

normal

long

difference, nominal , okay?

longer cable: increases decreases reading

extra cable removed?

4. WX temperature Pressure Humidity

5. Formatter time

6. formatter-GPS offset Counter Starts: GPS FMOUT

7. MASER -GPS offset Counter Starts: GPS MASER

8. VCs phase-cal Levels Frequencies

9. Local checks

10. Send "end" message with log location


11.0 Running "logpl"

This section describes the basics of running "logpl", a graphically oriented program for plotting

data from the log. This program is a more modern version of the old "logex" program (which is

still available). Please refer to the "logpl" manual available in PostScript form on the FS servers

in the doc sub-directory.

11.1. Interactive Use

"logpl" can be started from a shell prompt by just entering "logpl". The program uses the X

display, so the starting used must have access rights to the X display. If you are using X server

and open new window (C-S-W) and type "logpl" at the prompt, it should start okay.

11.1.1 Selecting a log file

If the FS is running, the log that is open will be selected automatically. If the FS isn't running

or if you want to use a different log, use the "File" menu item and select the "New Log File"

option. Enter the name of the log file you wish to examine. The directory will default to

"/usr2/log" and the extension to ".log".

11.1.2 Plotting Data

You can plot up to four "channels" of data simultaneously. The menu items "Channel1",

"Channel2", and so on are pull-down menus that let you select the data to be plotted for each

channel. The "Options" menu item allows you select whether the plot are superimposed or

shown separately and whether to connect the points with a line.

11.1.3 Entering New Parameters

You can enter new parameters to plot either by updating the "logpl.ctl" control file and

restarting "logpl" or by interactively using the "Options" menu item and selecting "Edit

Selections" options. Any changes that are made interactively are not preserved when "logpl" is

terminated.

11.1.4 Printing Plots

Once you have a plot you would like to print, you can print it as PostScript either to a printer or

to a file. Use the "File" menu item and select the "Print" option to print the file.

11.2 Non-interactive Use

"logpl" also has a non-interactive, or command mode, for automated processing. This is

particularly useful for automatically generating plot of ancillary data for inspection at shift

changes and after the experiment is over. To use "logpl" in command mode use the "-cmd"

command line option: "logpl -cmd". You can start processing from a command file

automatically by entering: "logpl -cfile xxx" where "xxx" is the name of the command file.

You can even select the log file to process on the command line by entering "logpl -log lll -

cfile xxx", where "lll" is the name of the log file.


11.2.1 Sample Experiment Delogging Command File

The following is a sample command file that will delog some general ancillary experiment

data. If you want to use this command file you should customize it for your station.

output=temp,overwrite

channel=1

command=wx/

parm=1

channel=2

command=wx/

parm=2

channel=3

command=wx/

parm=3

channel=4

command=cable/

parm=1

SCALE=.038308,.038365

plot

output=temp,append

channel=1

command=tsys1/

parm=9

scale=55,75

channel=2

command=tsys1/

parm=10

scale=50,70

channel=3

command=tsys2/

parm=7

scale=70,100

channel=4

select=0

plot

output=temp2,overwrite

command="

glist


11.2.2 Sample Output

The list below and first two following pages of plots show sample outputs from

"logpl" using the sample command file from the previous sub-section. The list of

comments is given here and the plots are shown after the page titled “Summary of Ops

Messages”, although that is not the order they would normally appear in.

------------------------------------------------

Filename: /usr2/log/ca034gc.log Selected data: Cmd=" Parm=7

9811117293172:" CA034 1998 GILCREEK A Gc

9811117293172:" A GILCREEK XYNS 7.3152 60.0 0 -86.0 86.0 60.0 0 -73.5 73.5 25.9 Gc 101

9811117293172:" Gc GILCREEK -2281547.29145 -1453645.07155 5756993.15313 40476601

9811117293172:" 101 MOJ-VLBA 1217640

9811117300862/newtape/"to continue, use label command"

9811117320970;"now doing cable length check

9811117323316;"adding extra cable

9811117334902;"removing extra cable

9811117345251;"shorter cable is higher measurement

9811117351006;"gps - maser 11.23 micros.

9811117352465;"gps - formout 11.8 micros.

9811117595001&preob/"nsource

9811118022476;" mostly clear skys

9811119010572;" scattered cumulus clouds



9811122150632;" mostly cloudy skys





9811203012995;"weather rpt.partly cloudy












9811215041480;" partly cloudy skys

9811216035476;" partly cloudy skys

9811217000980;" thin high stratus clouds

9811218003191;" thin high stratus clouds

9811218025818;"now doing cable length check

9811218031886;"adding extra cable

9811218041157;"removing extra cable

9811218050979;"shorter cable is higher measurement

9811218052061;"gps - maser 11.26 micros.

9811218053098;"gps - formout 11.8 micros.

9811218054115&unlod/"**dismount this tape now**" Glist: Time from

9811117293170 to 9811218065785. Data points listed: 684.

1998.237.17:22:55.12 1998.238.05:49:07.06 1998.238.18:15:19.01

1: Temperature3: Humidity

2: Pressure4: Cable−length

Station: GILCREEKFilename: /usr2/log/ca043gc.log

17.1

7.6

964.3

959.1

89.6

31.8

0.038365

0.038308

17.1

7.6

964.3

959.1

89.6

31.8

0.038365

0.038308

1998.237.17:22:55.12 1998.238.05:49:07.06 1998.238.18:15:19.01

1: Cmd=tsys1/ Parm=93: Cmd=tsys2/ Parm=7

2: Cmd=tsys1/ Parm=10Station: GILCREEKFilename: /usr2/log/ca043gc.log

75

55

70

50

100

70

75

55

70

50

100

70

75

55

70

50

100

70


12.0 Mark 5A Issues

12.1 When to change disk packs

During an experiment, the FS will use a pack until it is full. It will then attempt to switch to the

other bank. The full pack should be swapped with a fresh pack as soon as is practical. Great

urgency is obviously not important here since it should be hours until the new pack is needed.

Mounting a fresh pack as soon as possible will minimize the chances that data is lost if the

other pack has problems that causes it to be filled early. In that case data may be lost if there is

not an empty pack in the other bank. A problem such as this should be rare. However,

periodically monitoring the predicted time for next pack change may help to discover a

problem before it occurs. In addition, the output of the “mk5=get_stats?” commands in the

“postob_mk5a” procedure may show symptoms that may alert you to a problem. These

statistics should be periodically monitored. Both of these checks should be included in the

hourly, at least, checks.

Currently there is no automatic method to predict the time when a disk pack will be full and

ready to be changed. You can however get a pretty good idea by

comparing the GB count on the schedule summary print-out with the capacity of the disk and

using a little math. This technique is pretty accurate except in the case where there may be a

problem with a pack. In that case, the pack will fill sooner than expected. Additional

information is available from Monit-5 (Control-Shift-5) “Mark5 Remaining Capacity” display.

This display shows which pack is active, the remaining capacity of each pack (if it is known) in

three different units, and the UT time when this information was most recently determined for

each pack. The display of the remaining capacity of the active pack is updated during the set-up

for each observation. The remaining recording time in the current mode that is displayed is not

the remaining clock time. The clock time will depend on the recording duty cycle. For

example, for geodesy, the recording duty cycle is typically about 50%. So the active pack may

last about twice as long as the amount of time shown. This will vary from session to session

and station to station: your mileage may vary.

12.2 Changing disk packs

When the FS detects that the current pack is full, it will attempt to switch to the other pack. A

pop-up window will be displayed asking the operator to change the full pack (the display

includes information on which pack to remove). The full pack can be removed at anytime.

However, the Mark5 will become unresponsive to the FS for up to about 20 seconds when the

mount switch is turned to (lock) when the new pack is inserted. In order to avoid having this

impact the session, please be sure to do this only when the FS system will be quiescent for at

least 20 seconds, either between scans or while recording. In either case, it should be done 20

seconds or more before the next scheduled event. After completing the pack change, dismiss

the pop-up window to avoid confusion in the future.

12.3 Starting Mark5A and Recording Mark5A debug output


In order to help debug problems with the FS, Mark5A, and their interaction, we would like to

ask all stations to record the debug output generated by Mark5A. In order to this we

recommend that every time you start Mark5A, you use the following commands on the Mark5:

cd

script -af <name> Mark5A -m0 |& LLogr &

#(running of experiment occurs here)

EndM5 exit

The sequence above shows that you need to execute an "exit" (or Control-D) to terminate

"script" recording. Please be sure to terminate Mark5A and exit the "script" recording after

each experiment.

Please note the commands above assume you are using csh (or tcsh) as the shell.

You will need to adjust this accordingly if you are using a different shell, but you can always

start a csh (or a tcsh) and enter the commands as shown.

The "script" command shown above will cause all screen output to be put in a file with the

name "<name>" in your home directory. For "<name>", we suggest you use the name of

experiment with ".txt" appended, e.g. "r4174wz.txt" or if you are not running a schedule, you

might use the date for the part before the ".txt", e.g., "040224.txt".

If a problem is encountered with Mark5 control please send us the ".txt" file and the

corresponding FS log. This will aid greatly in debugging.

Periodically, you should delete old debug output files on the Mark5 so that the file system

13.0 Running "plotlog"

This section describes running "plotlog", an non-interactive program for generating plots from

log files. It was developed for analyzing the logs collected for the "overnite" and short-term

stability tests in "systests" . Basically, it plots any of the typically recorded ancillary data in a

log file. It plots everything it finds including: wx/, cable/, gps-fmout/, setcl#time/, tsys/,

receiver parameters, and phase-cal from the Mark IV decoder. Other items to be plotted can

easily be added, please let Ed ([email protected]) know if you want to include plots of

other data or want other command line options.

After the section and after the two example ‘logpl’ plots, there are five pages of sample output

from the ‘plotlog’ program.


13.1 Basic Command-line Use

The "plotlog" program is only run from a shell command-line or from a script. The basic usage

is:

cd /usr2/log

plotlog log_file

where log_file is the name of the log file, with the path if needed, including any suffix,

such as the typical ".log". This will display the plots on your terminal (press return to step

through the pages until your reach the last).

13.2 Plot Output to a File

The plots can be sent to a file for printing and/or other purposes, use:

cd /usr2/log

plotlog -f file/device log_file

where "log_file" is above, and "file" is the name of the output file, with the path if needed,

including any suffix such as the typical ".eps". The "device" should typically be vps (for

portrait orientation PostScript). You can use -f \? to be prompted for the device and at that point

there is an option to display a list of available devices.

See the "Sample Command and Output" subsection below for an example.

13.3 Command-line Options

There are several command-line options available to tailor the output. You can use "plotlog -h"

to display a help page with a full listing of options. You may not need any of the command-line

options, but the most useful ones are:

-c value -- delete certain cable values

This command allows you to delete the cable readings that were generated when the

extra cable was added for the sign check if "cablelong" wasn't used. If "value" is

negative, all points that far below the maximum value are deleted. If "value" is positive,

all points that far above the minimum value are deleted. The units for "value" are the

the ones on the vertical axis of the plot and so are affected by whether the "-r" option is

selected.

-p -- also plot phase-cal amplitude versus phase for available tones

-r -- show cable in raw units, rather than relative delay

-t value -- edit tsys values

All tsys points of zero or below and all tsys points greater than"value" are deleted.


-z value -- deletes PCal amplitudes less than "value"

See the "Sample Command and Output" subsection below for an example of the "-t" and "-c"

options.

13.4 Sample Command and Output

The command:

plotlog -t 500 -c -1000 -f /tmp/ca043gc.eps/vps ca043gc.log

was used to generate sample out from the same log as in section 11, for "logpl". The output is

included after the sample "logpl" plots at the end of this write-up. Typically, it should not be

necessary to use the "-t" or "-c" options, but they were needed for this old (1998) log. No phase

cal plots appear because there was no Mark IV decoder phase-cal data in the log.

13.5 Description of Plots

The subsection describes the basic features of "plotlog" output. The plots appear as up to six

plots per page. All time plots on all pages have the same horizontal time axis for easier

comparison. The reference time for "0 hours" appears in the title of the plot along with the

station name and the log name. No data editing is applied unless specified by command-line

options. The plots are auto-scaled to show all included data. All of the included data appears

within the box of the plot, never along the edges. Points that were not numeric or were

excluded by editing criteria appear along the top edge of _that_ plot. The pages are numbered.

On the last page the text "Last Page" is included so it is clear how many pages there should be.

Summary of Ops Messages

Send all messages to [email protected]. Please include the session name, station

name and type of message in the subject line. Please avoid the use of tab character and HTML

for these messages. The msg utility available now the FS has forms for Ready/Start/Stop

messages (In the initial version, the clock offset in the stop message must be entered as

comment).

1. Ready message

Comments

First source and time formatter to gps clock offset

WX: temp, pres, humid, sky conditions

Cable difference: longer cable makes reading larger/smaller change in microseconds and is/is

not nominal

Pointing values (as many as appropriate):

SEFD X/S Source Az/El Offset1 Offset2 707/705 Cas-A 352/34 -.004 .008

Tsys (x1/s/x2): 123/77/109

(The suggested format of the pointing and Tsys entries is shown above.)



1. Start message

After starting, report source and start time

2. Stop message

Comments

End formatter to gps clock offset list of problem scans

Module inventory on site

Observation end time

All module shipping information (VSN, AWB, shipper) should be placed in

TRACK/PACKTRACK. Please wait until you have the AWB numbers available to do

this unless it would cause an unreasonable delay. This item is not part of the message,

just a reminder to enter the information into TRACK/PACKTRACK.

3. Wrap-up message

Optionally this information may be included in the Stop message, you can of course send as

many of these follow-up messages as you need. The wrap-up message may also contain log

placement information, any additional shipping information.

Additional information about problems during the session should be sent as a wrap-up message

at any time after discovery post observing.

VGOS Operators Notes

2017-03-07

Contents

1 Introduction 1

2 Pre-setup 2A DRUDG experiment files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2

3 Experiment setup 2A Verify NTP is sync’d . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2B Start FS log file . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 2C Check RDBE Status . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3D Check Mark 6 Status . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3E Check MCI Status . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3F Mount Mark 6 Modules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 3G Verify RDBE time, offsets, and VDIF epochs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 4H Initialize pointing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 5I Set mode and attenuators . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6J Check RDBEs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 6K Check pointing . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7L Make test recording . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 7

4 Start experiment 8A Start non-FS multi-cast logging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8B Send “Ready” message . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 8C Start schedule . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9D Send “Start” message . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

5 Monitor experiment 9A Monitor scan_check . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9B Check RDBE Monitor . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9

6 Post experiment 9A Stop the schedule . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 9B Stop multicast logging . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10C Check pointing and SEFDs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10D Send “End” message . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 10E Send test scan data files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11F Remove the module for shipping . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 11G Transfer log file . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 12

7 Appendix 13A Setting Up Password-less SSH . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13B Setting Up Password-less Log Transfers . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13C Manually uploading log files . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 13D Manually processing schedules . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14E Schedule rotation . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14F Module conditioning . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 14G Setup Field System PC from Cold start . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15H Setup of RDBEs from a cold start . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 15I Set-up of Mark 6 server from a cold start . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 16J Setup MCI server . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 17K Connecting RDBE IF inputs . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . . 18

1. IntroductionThese notes cover basic procedures for running VGOS experiment operations with the Field System (FS). Theyare described in a logical order for the tasks that need to be completed. If another order of operations make

1

more sense for some locale that is fine of course.A basic assumption is that all the equipment is already up and running. The procedures check this and refer tothe appendix for set-up instructions for a device if it isn’t already running. In addition to these setup procedures,the appendix also contains instructions for some other useful operations.As written here, the procedures are mostly station independent. However, some station dependencies arenoted. These only cover GGAO and KPGO for now but Westford can, and will, be added. Some of the stationdependencies can be removed by writing some additional scripts that hid the dependencies. This is also a goodidea because the scripts can reduce the complexity of the operations.

EH: The etransfer instructions need to be checked.

2. Pre-setupA. DRUDG experiment files

This step is first for two reasons. First, the schedule file is typically available before the day of the experiment. Secondly, the number of modules in a recording group may eventually depend on the experiment. When that is the case, the number needed will conveyed through the listing that is made in this step.To create the station specific SNAP and procedure files from the session schedule, fetch the schedule from IVS and DRUDG it by entering, in a FS PC shell, the commandfesh -d <sched>

where <sched> is the schedule name, eg v16033. If you have manually received the schedule, follow the instruc-tions in the Manually processing schedules section of the appendix.Along with the .snp and .prc files for the FS, this will create the listings file <sched><stn id>.lst in /usr2/sched which shows a human readable summary of the schedule. Here <stn id> is the two letter station code, eg “gs” for GGAO. To view this on the FS PC, open in a text viewer; eg, in a FS shellless /usr2/sched/<sched><stn id>.lst

Review the information in this listings file to ensure you have sufficient storage for the observation.

3. Experiment setupA. Verify NTP is sync’dIn the FS enter:

check_ntp

The output is of the form

remote refid st t when poll reach delay offset jitter==============================================================================*192.168.1.20 18.26.4.105 2 u 444 1024 377 0.208 -0.336 0.396+192.168.1.21 18.26.4.105 2 u 167 1024 377 0.215 -0.822 0.153

The offsets should be small and there must be a server with an asterisk * in the first column. It may take a fewminutes to get an *.

B. Start FS log file

In the FS, open the experiment log so the set-up will recorded in that log:log=<schedule><stn id> # eg v16033gs

2

C. Check RDBE Status

From the Field System, check the RDBEsrdbe_status

The response values should be 0x0f41.

Old Server: The response values should be 0x0941.

If the response is to not correct for any RDBEs, refer to Setup of RDBEs from a cold start in the appendix.

D. Check Mark 6 Status

From the Field System, check the Mark6smk6=dts_id?

You should receive a sensible response similar to!dts_id?0:Mark6-4605:1.0.24-1:1.2;

If the response is to not correct for any Mark6s, refer to Set-up of Mark 6 server from a cold start in theappendix.

E. Check MCI Status

This is specific to GGAO.

You can test whether this is needed by using the FS SNAP procedure:dewar

This is specific to KPGO.cryo

If it is working, you will see the readouts for the vacuum pressure and 20K and 70K stages. If not, see the SetupMCI server section in the appendix.

F. Mount Mark 6 Modules

This step is included just after initial verification that all the devices are working, but before the more detailedpre-experiment checks. This allows any issues with getting the modules mounted resolved as soon as possible.However, these steps can be done as part of the making the test recording if that is preferred. All commandsare entered into the FS unless otherwise noted.1. This is to help with debugging, display and clear the Mark 6 message queue:

mk6=msg?

If an unexplained error happens during the following procedure, please use this command again to get moreinformation.

2. Initialize module; create, mount, and open moduleCheck status:mk6=mstat?all

After the two fields: return code and cplane status (hopefully mstat?0:0) there are 10 fields per group:group:slot:eMSN:#disks found:#disks nominal:free space:total space:status1:status2:type

It may be easier to read if individual groups are queried; eg. for group 1:mk6=mstat?1

3

If the module has already been initialized, ie status1 is initialized, and the data is no longer needed (becertain first), erase it:mk6=group=unprotect:<group>mk6=group=erase:<group>

If the module has not been initialized (status1 is “unknown” and no eMSN?), initialize it:mk6=mod_init=<slot#>:<#disks>:<MSN>:<type>:<new>

For examplemk6=mod_init=1:8:HAY%0001:sg:new

Note: due to a current incompatibility, the FS–Mark 6 connection will timeout during long runningcommands such as this.

Until this is fixed, you can optionally run this command directly on the Mark 6 from a shell promptwithssh root@mark6ada_clientmod_init=<slot#>:<#disks>:<MSN>:<type>:<new>;

note the final semicolon is necesseary in da_client but is automatically added by the FS.

Create, open and mount the group:mk6=group=new:<slots>mk6=group=mount:<slots>mk6=group=open:<slots>

(Slots is a list of slot numbers included in the group, without any seperators, eg <slots>=12)To query if the group is created properly:mk6=group?

Print out should have the group number at the end. If it is -, something has gone wrong.

G. Verify RDBE time, offsets, and VDIF epochs

In the FS, check RDBE time, offsets, and VDIF epochs:time

This will display the pps_offset, dot, and gps_offset:2016.320.18:11:39.28/rdbed/!dbe_pps_offset?0:-1.953124995e-08;2016.320.18:11:39.28/rdbec/!dbe_pps_offset?0:-1.953124995e-08;2016.320.18:11:39.29/rdbea/!dbe_pps_offset?0:-1.953124995e-08;2016.320.18:11:39.29/rdbeb/!dbe_pps_offset?0:-1.953124995e-08;2016.320.18:11:39.29/rdbeb/!dbe_dot?0:2016-320-18-11-39.296s:-0.007s:33;2016.320.18:11:39.29/rdbea/!dbe_dot?0:2016-320-18-11-39.296s:-0.018s:33;2016.320.18:11:39.29/rdbed/!dbe_dot?0:2016-320-18-11-39.296s:-0.024s:33;2016.320.18:11:39.29/rdbec/!dbe_dot?0:2016-320-18-11-39.296s:0.000s:33;2016.320.18:11:39.29/rdbeb/!dbe_gps_offset?0:-3.758203125e-05;2016.320.18:11:39.29/rdbed/!dbe_gps_offset?0:-3.758203125e-05;2016.320.18:11:39.29/rdbea/!dbe_gps_offset?0:-3.758203125e-05;2016.320.18:11:39.29/rdbec/!dbe_gps_offset?0:-3.758203125e-05;

The offsets should be small (GPS typically ± a few tens of µs, PPS less than ±0.1 µs) and the DOT times shouldbe the same the FS log timestamps, within about 0.1 seconds. The third field of ‘dot’ output is the VDIF epoch.All RDBEs must have the same value, (33 in this case). The VDIF epochs of all RDBEs are also shown in theRDBE status window.

For old server, the VDIF epochs are not displayed

4

If any of the DOT times are not correct, the ones that are wrong must be set with ‘fmset’. To do this from theFS console, press <Control><Shift>T to start fmset, or typefmset

in an FS PC shell.You can select the RDBE to set by letter: a, b, c, or d.With that RDBE’s time being displayed, verify that the time is correct by comparing it to the FS/Computer time.If it is off by a lot, use ‘.’ to get it close — within a few seconds. Once it is close, you can use ‘+’ and/or ‘-’ toincrement and/or decrement the RDBE by a second at time until it agrees with the FS time.If the PPSoffset is greater inmagnitude than ±1e-7 (±0.1 µs) for an RDBE, it must be resync’d. You can try usingthe ‘s’ command in ‘fmset’ for each RDBE that has too large an offset. This command will take approximately45 seconds per RDBE to complete. If as a result, the PPS offset comes within the tolerance, please check thatall the RDBEs are sending data withmk6in

If an interface is not showing receipt of data, reinitilize the corresponding RDBE with:rdbe_init<id>

where <id>=a, b, c, or d for the interface that is not recieving data, eth2, eth3, eth4, or eth5 respec-tively.If this does not solve the data transmission problem. or the ‘s’ did not make the offset small enough, restartthe RDBE, see Setup of RDBEs from a cold start in the appendix.

For the old server if the PPS offset is too large, you will need to restart the RDBE to re-sync, see Setupof RDBEs from a cold start in the appendix.

If the displayed VDIF epochs are not the same, use the ‘;’ command for each RDBE to set the epoch to thenominal one.

For the old server, the only way to verify this is to note that the Mark 6 will not record a test scanwhen you make one (but there could be other causes besides this one). If this happens and you candetermine which RDBEs do not have the current nominal VDIF epoch set its time explicitly. If you don’tknow which is wrong just set them all explicitly. Use the procedure described below for after June 30or December 31 to set the time(s).

If this is the first experiment since December 31 and June 30, and the RDBEs have not had their epoches resetsince that date, they should be reset. Use the ‘;’ command in ‘fmset’ for each RDBE to set it to the nominalVDIF epoch. The third field in the ‘dot’ output above must be the same for all the RDBEs. The VDIF epochs ofall RDBEs are also shown in the RDBE status window.

For the old server, this requires setting the time explicitly for each RDBE with ‘fmset’ even if it lookscorrect. Use at least one of ‘.’, ‘+’, ‘-’, or ‘=’ commands and then it is necessary to verify/set the timefor each RDBE. The third field of the ‘dot’ output above will be missing for the old server and the VDIFepoch will not be in the RDBE status window.

If any changes were necessary due to the above considerations, check the values again with:time

to verify they are correct. If not, follow the above instructions again and/or seek assistance.

H. Initialize pointing

In the FS, initialize pointing configuration and send antenna to a test source:proc=pointinitpcasa

The following sources are the most reliable for these small antennas:

5

Source Approximate L.S.T. of transitTaurus A 05:30

Virgo A 12:30Cygnus A 20:00

Cas A 23:30

This is KPGO specific:

verify Az and El for source are acceptableantenna=operate

If not, try a different source first.

This following information is site specific.

Local apparent sidereal time (L.A.S.T) is displayed in the antenna monitor window (monan) at GGAO and KPGO.Cas A is always up at GGAO and Westford, but another source may be more appropriate at times.

I. Set mode and attenuators

While waiting for the antenna to move to the test source, setup the experiment mode and adjust attenuators.In the FS,proc=<schedule><stn id> # eg. 'v16033gs'setupbbifdbbmk6bbauto # sets the attenuatorsproc=point

J. Check RDBEs

Locate the RDBE Monitor window (monit6) or start it by pressing <Control><Shift>6. Noting that for somefields the display switches between IF0 and IF1 every second, check for each RDBE that:1. DOT ticking and correct time2. All RDBEs have the same epoch

Not applicable to old server

3. DOT2GPS value small (±a few µs) and stable (varies by 0.1 µs or less)4. DOT2PPS value small (±0.1 µs) and stable (varies by 0.004 µs or less).5. RMS values close to 20. They may be higher if the antenna has reached the source since the “auto” command

above. They may be higher or lower if the elevation has changed significantly since the “auto” or there isvariable RFI.

For the old server, RMS values close to 32.

6. Tsys IF0 and IF1 about 50-100, may be jumping a bit7. Phase-cal amplitude about 10-100, phase stable to within a few degreesLeave the window open for later monitoring.

EH: Is this still needed?

Check multicast for all 4 bands in FS shell prompt:mon<id>

where <id>=a, b, c, or d (eg. mona etc.)

Check RDBE data connectons

6

rdbe=data_connect?

and verify that band a,b,c,and d equal 0,1,2, and 3, respectively.

K. Check pointing

Check that the antenna is now on the source we selected earlier with the FS commandonsource

The result should be TRACKING. If the antenna status is still SLEWING wait until you see an onsource message inthe FS log window or onsource indicates tracking.Once the antenna is onsource, start the pointing check withfivept

This will take a few minutes. Once complete fivpt will give you output in the form:Az El xEl_offs El_offs

xoffset 99.4469 30.8190 0.01417 -0.00806 0.00452 0.00801 1 1 01d0 virgoa

The xEl_offs and El_off values (ie. the 3rd and 4th columns) are the beam spaces offsets of the pointing fit.The absolute value of these should be less that ~0.02 degrees in each coordinate. There should also be theflags ‘1 1’ in the 3rd and 4th columns from the end.Next, measure the SEFDs on test sourceonoff

This will also take a few minutes. Once complete onoff will give you output in the form:source Az El De I P Center Comp Tsys SEFD Tcal(j) Tcal(r)

VAL virgoa 170.9 63.0 15a0 1 l 3016.40 0.9943 52.03 2895.6 55.657 1.67VAL virgoa 170.9 63.0 15a1 2 r 3016.40 1.0088 47.93 2549.8 53.201 1.60VAL virgoa 170.9 63.0 15b0 3 l 5256.40 0.9946 49.58 2742.3 55.306 1.66VAL virgoa 170.9 63.0 15b1 4 r 5256.40 1.0148 41.57 2549.6 61.331 1.84VAL virgoa 170.9 63.0 15c0 5 l 6376.40 0.9831 42.57 2294.2 53.891 1.62VAL virgoa 170.9 63.0 15c1 6 r 6376.40 0.9862 44.09 2248.1 50.992 1.53VAL virgoa 170.9 63.0 15d0 7 l 10216.40 1.0121 51.91 3009.5 57.979 1.74VAL virgoa 170.9 63.0 15d1 8 r 10216.40 0.9870 53.64 3084.2 57.496 1.72

source Az El De I P Center Comp Tsys SEFD Tcal(j) Tcal(r)

Verify SEFDs for eight bands are reasonable. They should be in the range ~2000-3000.Finally, zero the offsets:azeloff=0d,0d

(note there is no space between arguments)

L. Make test recording

1. Check Mark 6 inputs withmk6in

which will show the Gb/s by interface in the FS log. For example, a rate of 2 Gb/s should should look like#popen#mk6in/eth2 2.078 eth3 2.079 eth4 2.079 eth5 2.079 Gb/s

If one or more interfaces are not showing the approximate nominal data rate (initially 2 Gb/s per interface),it is likely that the corresponding RDBEs needs to be reconfigured.To see more details of the Ethernet ports state for the Mk6, usemk6=input_stream?

Sample output:

7

mk6a/!input_stream?0:0:rdbeB:vdif:8224:42:66:eth3:127.0.0.1:12000:0:rdbeC:vdif:8224:42:66:eth4:127.0.0.1:12000:0:rdbeA:vdif:8224:42:66:eth2:127.0.0.1:12000:0:rdbeD:vdif:8224:42:66:eth5:127.0.0.1:12000:0;

which shows rdbeA going to eth2, rdbeB going to eth3, rdbeC going to eth4, and rdbeD going to eth5.2. The modules and groups should have been set-up already. If not refer to Mount Mark 6 Modules above.3. In FS, record some test data:

mk6=record=on:30:30

verify that lights on the mk6 flash appropriately. You can check recording status with:mk6=record?

It should progress starting as “recording”, then transitioning to “off”.If the status stays “pending”, it may be that not all the RDBEs are sending data. You can check this by usingthe FS SNAP proceduremk6in

as shown in step #1 above.If all interfaces are receiving data at the correct rate, it may be that the VDIF epochs of all the RDBEs don’tagree. See Verify RDBE time, offsets, and VDIF epochs for detail on how to verify/set.You can also check if the disk is full withmk6=rtime?

4. Once recording ends, check quality:mk6=scan_check?

Results should show vdif, the time when recording was started, 30 seconds of data, 30 GB of data with an8 Gbps data rate.

EH: data rate will eventually go to 16 and 32 Gbps.

4. Start experimentA. Start non-FS multi-cast logging

EH: Once we have InfluxDB logging of data, maybe we can get rid of this logging,

From the FS enter:start_mlog

AB: Doesn’t seem to work at Wf

If there are no errors reported and the “Done” message is printed the logging has been started.

B. Send “Ready” message

From FS console window enter Control-Shift-G.

EH: Jason will eventually move vgos-msg-gui.py to the FS machines so we can have better function-ality. Maybe the placement of the window should be controlled locally by .Xresources.

At this point a GUI window should pop up. Enter the session name, station code (lower case) and select thetype of message from the drop down list.

• Click the update values button. This collects the information in real time and the SEFDs from the pointingcheck in the log file.

• Complete the maser offset value by looking at the maser counter in the maser room.

8

• In the “to” email address field, send it to [email protected]

• Enter a brief comment, include weather information.• Click the send message button when finished.

C. Start schedule

In a Linux shell (xterm), look at the list file <schedule><stn id>.lst created in the DRUDG step (eg. v16033gs.lst). Find the first observation and note line number ‘nnn’ after scan name at start of line.

Now, in the FS, start schedule:schedule=<session><stn id>,#<nnn>

(Note: the pound sign (#) is required and there should be no spaces in the command)

D. Send “Start” message

Send “Start” message using the same procedure as in Send “Ready” message.

5. Monitor experimentA. Monitor scan_check

To display scan_check results as they come in (and the old ones so far) open the ‘scnch’ window (Control><Shift>K).

Results should show vdif, reasonable record start time, about equal seconds and GBs of data (typically 30+),and 8 Gbps data rate. Be aware scan_checks occasionally fails and the data is okay.Position and size window for convenient viewing, new output will follow any changed size. You can stop thiswith <Control>-C

EH: The placement and size of the window can be controlled by .Xresources.

B. Check RDBE Monitor

Check the display for reasonable values periodically, note some fields alternater between IF0 and IF1 everysecond:1. DOT ticking and correct time2. VDIF epoches for all RDBEs agree, if there is any disagreement some will be in inverse video

For old server, not applicable.3. DOT2GPS value small (±a few µs) and stable (varies by 0.1 µs or less).4. DOT2PPS value small (±0.1 µs) and stable (varies by 0.004 µs or less).5. While recording RMS values are close to 20 for the new server, sometime RFI can cause the value to be off,

but it should always be between 10 and 40 during recording. If valeus are outside the nominal ragnget theywill be shown in inverse video.

For the old server, RMS values close to 32 and inverse video is not used.6. Tsys IF0 and IF1 about 50-100 (may lower at Wf due to use of a preliminary cal value), may be jumping a bit7. Phase-cal amplitude about 10-100, phase stable to within a few degrees.

6. Post experimentA. Stop the schedule

In the FS, runschedule=

9

B. Stop multicast logging

In the FS, runstop_mlog

C. Check pointing and SEFDs

Select procedure ‘point’ libraryproc=point

If the FS has been restarted since the initial check, you will need to re-initialize the pointing set-upinitp

Site Specific for KPGO:

antenna=off (to allow verification of Az and El for source before antenna moves)

Try Cas-A as a source:casa

Site Specific for KPGO:

If necessary, try other sources from table in Initialize pointing until one with a good position is found,then:

antenna=operate (restart antenna)

If you re-ran ‘initp’ above, you will need to restore the experiment set-up:proc=<schedule><stn id> # eg. 'v16033gs'setupbbifdbbmk6bbautoproc=point

Wait until the antenna is on source. You can either watch the log or check withonsource

The result should be “tracking”.As in the Check pointing section in pre-experiment, run a pointing checkfivept

and check the “xoffset” offset values are small. Then check the SEFDsonoff

and verify SEFDs for eight bands are reasonable, ~2000-3000.Finally, zero the offsetsazeloff=0d,0d

Site specific for KPGO:

source=stow (wait until stow is reached) antenna=off

D. Send “End” message

Send “End” message using the same procedure as in Send “Ready” message. Include details such as the stoptime and the current weather conditions on-site.

10

E. Send test scan data files

Chris: This section is completely different for KPGO, due to our e-transfer Mk6 being a different unitthan our operational Mk6. Also for actually sending entire experiments not just test scans. Below theoriginal steps provided in this procedure are the KPGO site specific steps highlighted.

In a terminal, log in to the Mark 6

EH: this is the part I know the least about and I suspect it different for different stations, maybe usingsomething besides “gather”, have tried it at GGAO?

ssh mark6agator <group> <filename>.vdif ~/dqa –d <filename>.vdifscp <filename>_*.vdif evlbi1.haystack.mit.edu:/data-st12/vgos/<exp>/

EH: Maybe put into a script, or something, to minimize typing? As it is, it is definitely too much typing

F. Remove the module for shipping

In the FSmk6=group=close:<slots>mk6=group=unmount:<slots>

Before removing, check the modules are unmounted with

key off disk before doing mk6=mstat?all

mk6=mstat?all

Chris: KPGO site specific send test scan and e-transfer procedure

Close and unmount disk module(s) and prepare for e-transferring a scan or experiment.mk6=group=close:<slots>mk6=group=unmount:<slots>

turn keys off, remove module(s)mk6=mstat?all

(to clear module info and check the modules are unmounted)Insert Mark6 modules into the e-tranfer Mark6From the da-client mount the modules and verify all disks are seen:da-clientgroup=mount:<slots>;mstat?all;

(if you get “6:0:1” restart cplane)group=open:<slots> list?

From another xterm window gather the scan(s) to your RAID disk, and de-thread if necessary:For test scan that needs to be de-threaded:gator <slots> <scan name>.vdif /mnt/raiddqa -d <scan name>.vdif

(this will create 4 files with thread ID on scan name)For scans where you intend to transfer the entire experiment use gather464:gator -t <slots> "scan name".vdif /mnt/raid

11

Start tsunami server specifying the scans of the session to transfertsunamid <scan_name>_*.vdif

You will see the available scans to be pulledAt another xterm window (in “oper”, not “root”)Ssh to Haystack storage nodes:ssh evlbi1.haystack.mit.edu (password is oper password)cd /data-st12/vgos

Run tsunami, setting the transfer rate, error free, and connecting back to your machinetsunami set rate 100M set error 0 connect 146.88.148.18

Make sure needed files are there to be pulleddir

Pull filesget *

Once transfer is complete exit tsunami client to get prompt backexitls <scan_name>* (verify all scans were copied)

After last scan has copied logoutlogout<Ctrl-C> (to quit server)

From da-client unmount the disk and prepare for shippinggroup=unmount:<slots>;

turn keys off, remove modulesmk6=mstat?all

(clears module info and checks the modules are unmounted)

G. Transfer log file

This may be done before transferring the test scan.

In FS, close experiment log:log=station

In a terminal, copy the log to CDDIS and Haystack with plog. If you are transferring the most recent log you canuseplog -l

Otherwise useplot <session> # v16033

Or if the log file does not conform to the standard naming conventionplog <log file path> # eg /usr2/log/v16033gs.log

If this is not successful, see Manually uploading log files in the appendix

12

7. AppendixA. Setting Up Password-less SSH

It is convenient to setup password-less login for local devices from the Field System PC. You can do this withSSH using public-key cryptography. To generate public/private key pair with SSH (if you don’t already have one),runssh-keygen

Accept the defaults and enter a blank password when prompted.For each computer you want to enable password-less login, append your public key to .ssh/authorized_keyson the remote host. On a recent versions of the Field System OS (i.e., FSL9 based on Debian Wheezy) use thecommand (use the target host node name or IP address in place of $host):ssh-copy-id $host

If this is not available usecat ~/.ssh/id_rsa.pub | ssh $host 'cat >> ~/.ssh/authorized_keys'

If you do not wish to have completely password-less login, an alternative is to encrypt your ssh key with apassword and use ssh-agent to unlock it for your session. The upshot is you still have the convenience ofpassword-less login, you just have to enter your password once after you login to the FS computer. Howeverthis is not recommended for devices that FS procedures use ssh to connect to, including, RDBEs, Mark 6s, MCI,and backend-PCs, which will need to have no prompt for smooth operations.This is also more secure since the ssh key is encrypted on disk and if anyone ever takes your key, they can notgain access to your systems.This is a good idea for remote terminals, although is slightly more cumbersome for local access.To encrypt your private key, enter a password when you generate it. To encrypt an old key, or change its pass-word, usessh-keygen -p -f ~/.ssh/id_rsa

The process for adding your public key to a login to a remote host is the same as above.Now, when you want to use your ssh key, add it to your ssh-agent withssh-add ~/.ssh/id_rsa

This will decrypt your private key and allow any ssh clients the current login session to use it without a password.

B. Setting Up Password-less Log Transfers

EH: replace with new CDDIS transfer set-up procedure, use URL reference

Currently we are using FTP to transfer log files to CDDIS. This is will change in the future. For now, add thefollowing line to the ~/.netrc file:machine cddisin.gsfc.nasa.gov login <username> password <password>

replacing the appropriate fields with your username and password.

C. Manually uploading log files

EH: must be re-written for new CDDIS procedure

In a terminal, copy the log to CDDIScd /usr2/logftp cddisin.gsfc.nasa.govuser <your cddisin username>password <your cddisin password>put <session><stn id>.log # eg 'v16033gs'quit

13

And to Haystack:scp <session><stn id>.log evlbi1.haystack.mit.edu:/data-st12/vgos/logs

D. Manually processing schedules

1. Put schedule in /usr2/sched on FS PC2. Run drudg, from FS PC shell:

cd /usr2/scheddrudg <schedule>.skd

Now, in drudg, give the following commands (ignoring text after # and filling in the variables)<stn id> # the two-letter station id (eg. 'gs' at GGAO)3 # make .snp12 # make .prc9 # change printer output destination<schedule><stn id>.lst # destination file, eg 'v16033gs.lst'

# three more <Return>s5 # print summary0 # exit DRUDG

AB: Should use /exper/ directory?

E. Schedule rotation

1. Start DRUDG with original schedule2. Pick option 10 in DRUDG.3. Specify the full fine name for the output file, i.e., include the .skd. I suggest you call it hYYDDD.skd. The “h”

is to avoid confusing it with real schedules.4. Pick the start time. YYYY MM DD HH MM SS5. Pick the duration — usually 24 hours6. End DRUDG7. Restart DRUDG with the new file to make the normal output (or skip step #6 reselect schedule, option #8)

F. Module conditioning

1. Load modules and enter da-clientssh mark6a da-client

2. In da-client, initialize the modules with the same mod_init command used for experiment set up:mod_init=<slot#>:<#disks>:<MSN>:<type>:<new>;

3. Create a new group with the modules you want to condition. If more than 1 module is being conditioned,group them together.group=new:WXYZ;group=mount:WXYZ;group=open:WXYZ;

WXYZ=slot 1, 2, 3, or 4. Only enter slots with modules in them.4. Check to the status. It should say “open:ready” for the modules included in the group.

mstat?all;

5. Leave da-client and navigate to bin.<Control>+Ccd /home/oper/bin

14

6. Run the hammer script. After, all 8 lights on each module should be lit.nohup hammer.sh &

7. To break the group, do the mod_init command on each module and reassign groups. If recording on all themodules simultaneously, no further action is needed before an observation besides a test recording.

G. Setup Field System PC from Cold start• Start FS computer• Login as user oper• Check NTP by runningntpq -np

The output is of the formremote refid st t when poll reach delay offset jitter

==============================================================================*192.168.1.20 18.26.4.105 2 u 444 1024 377 0.208 -0.336 0.396+192.168.1.21 18.26.4.105 2 u 167 1024 377 0.215 -0.822 0.153

The offsets should be small and there must be a server with an asterisk * in the first column. It may take afew minutes to get an *.

• Other local devices that use NTP (antenna, RDBE, Mark6, etc) can now be started• Start the Field System in the login shell

fs

H. Setup of RDBEs from a cold startPower up RDBEs

Use the power switch to start or cycle the power of each RDBE to be started.

New server:

Check the status of all RDBEs with the FS commandrdbe_status

When the RDBEs respond with a status value 0x0f41, skip to Check/Set RDBE times

Start RDBE server

From FS PC shell prompt, login to each RDBE:ssh root@rdbe<id>

use <id>=a, b, c, or d to log into RDBE-<id>.Then, on each RDBE runrbinnohup ./rdbe_server 5000 6 &exit

Repeat this for each RDBE that has been restarted. You can verify when all the RDBEs have started from the FSwith:rdbe_status

There will be an error for each RDBE that is not ready. When all RDBEs respond with a status value 0x0941,proceed to the next step.

15

Load Firmware

To load the firmware on all RDBEs, use the FS command:rdbe_fpga

If you want to load the firmware individually for RDBE-<id>, you can use the FS commandrdbe_fpga<id>

Again, use <id>=a, b, c, or d as necessary.There will be an error for each RDBE, since it will not respond right away and will time-out. You verify when thisis finished from the FS with again usingrdbe_status

This time, the RDBEs should respond with status 0x0f01

EH: is 0x0f01 correct?

When all status values reach the correct value, proceed to next step

Configure RDBEs

To initialize the configuration on all RDBEs, use the FS command:rdbe_init

You should get four “success” messages.If you need initialize an RDBE individually use, the FS commandrdbe_init<id>

You should get a “success” message.

Check/Set RDBE times

It is necessary to check/set the time with fmset for an RDBE every time it is restarted. The time only needs tobe set if it is not correct already.To do this from the FS console, press <Control><Shift>T to start fmset, or typefmset

in an FS PC shell.You can select the RDBE to set by letter: a, b, c, or d.With that RDBE’s time being displayed, verify that the time is correct at the one second level (± a few tenths) bycomparing it to the FS/Computer time. If it is off by a lot, use “.” to get it close, within a few seconds. Once it isclose, you can use + and/or - to increment and/or decrement the RDBE by a second at time until it agrees withthe FS time.Be sure to exit with <Escape>.

I. Set-up of Mark 6 server from a cold start

Use the power switch to start or cycle the power of each Mark 6 to be started.

Check Mark 6 connection (New server)

From the Field System, check the Mark 6 connectionmk6=dts_id?

You should receive a sensible response similar to!dts_id?0:Mark6-4605:1.0.24-1:1.2;

If so, you can skip the rest of this section.

16

Starting Mark 6 servers

If you receive an error, check that the Mark 6 servers are running. The programs cplane and dplane need to berunning on the Mark 6. These should startup after boot.To check check if they are running performssh root@mark6aps aux | grep plane

If they are not, start them/etc/init.d/dplane start/etc/init.d/cplane start

J. Setup MCI server

This is specific to GGAO.

GGAO should be set-up agree with KPGO FS approach below.

You can test whether this is needed by using the FS SNAP procedure:dewar

If it is working, you will see the readouts for the 20K and 70K stages. If not, or if more MCI parameters aredesired, use the following from a shell window, login to the MCI computer and run the MCI clientssh mci./tcpip_client mci 10000

a prompt should come up. To display all the MCI data use the commandmci_data?

If the server is not running, start it with./startmciserver

This is specific to Westford

Log in to the MCI node PC from a new window on the FSPCssh 192.52.63.139

Confirm the server is running (called ‘fenode_server’)ps aux |grep server

If it is not running, start as followscd node-software/V0/./mci_fenodesrvr 192.52.63.139

At this point you should see data points all the way through DI345 scroll once per minute. If you close thiswindow, the server will quit.

This is site specific to KPGO

You can test whether this is needed by using the FS SNAP procedure:This is specific to KPGO.cryo

If it is working, you will see the readouts for the vacuum, 20K, and 70K stages. If not, start the server from theFS:startmci

17

If ‘cryo’ still doesn’t work, then use log into the Hub PC in new xterm window of FSssh oper@hubpcps aux|grep mci (to see if mci server is running)startmciserver (to start the server if not running)

To display more MCI date from the FS, enter:mci_data

If that doesn’t work, log into the Backend PC in new xterm window on FSssh oper@backend-pcmci_client.py 128.171.102.237 5000 (opens mci client on backend pc)mci_data? (displays all mci data points current state including dewar temperatures)

K. Connecting RDBE IF inputs

To avoid possibly overloading, and damaging, the RDBE analog-to-digital (A2D) converters used to sample thesignal, it is recommended to make sure that the attenuators are at the maximum setting (31.5)c when connect-ing cables to the IF inputs. Use the command:rdbe_attenX=<if>,31.5

where:‘X’ is the RDBE ‘a’, ‘b’, ‘c’, ‘d’ or null for all. ‘<if>’ is the IF channel ‘0’, ‘1’, or ‘both’ (or null) for bothfor example to set IF1 attenuator for RDBE-B to the maximum use:rdbe_attenb=1,31.5

to set both IF attenuators for RDBE-C to the maximum use:rdbe_attenc=,31.5

or to set all IF attenuators for all RDBEs to the maximum, use:rdbe_atten=,31.5

18

Westford VGOS Setup Checklist

123456789

10111213141516171819202122232425262728293031323334353637383940414243444546

A BDate ____________

fesh -d <sched> Automatically gets files and creates session files

check_ntp Verify the NTP is sync’d

log=<schedule name><station id> vt7011wf

rdbe_status 0x0f41

mk6=dts_id? !dts_id?0:Mark6-4605:1.0.24-1:1.2;

Confirm MCI is running by viewing display and confirming clock is running

mk6=msg? clears mark6 message que

mk6=mstat?1 Query for group 1

If needed erase module:

mk6=group=unprotect:<group>

mk6=group=erase:<group>

If needed initialize module:

mk6=mod_init=<slot>:<#disks>:<msn>:<type>:new mk6=mod_init=1:8:HAY%0001:sg:new

Create, Mount, and Open the module for operations

mk6=group=new:<slots> mk6=group=new:1

mk6=group=mount:<slots>

mk6=group=open:<slots>

mk6=group? Confirms if group is created properly

time This will display the RDBE pps,dot, and gps

If there needs to be a time correction use the fmset command

mk6in then RDBE=data_connect? Confirms RDBE data is streaming and cabling correct If needed to re-initialize the RDBE:

rdbe_init<id> id is either a,b,c,d

time run this to confirm status of any changes

Run mona,monb,monc,mond from new window to confirm phase cal tones then Start pointing checks:

proc=point

initp

casa

proc=<schedule name><station id> vt7011wf

setupbb

ifdbb

mk6bb

auto

proc=point

onsource

fivept Westford normal offsets less than .025 degrees

onoff Sefd numbers 2000-5000 are normal

azeloff=0d,0d

mk6=input_stream?

mk6=record=on:30:30

mk6=record? Do this command while recording to show status

mk6=rtime? Displays recording space left on disk

mk6=scan_check?start_mlog This starts the multicast data

Send Ready Message

Westford VGOS Setup Checklist

47484950515253545556575859606162636465666768697071727374757677787980818283848586

A BStartremote This starts the Haystack remote terminal monitorSchedule=<session><station id>,#<nnn>

Send Start message Then run Cable cal procedure and record valueControl Shift K This opens a window displaying scan_checks from active log file

Monitor session and enter hourly WX, and operational comments

End of Session:

schedule=

stop_mlog

Send Stop message

Do pointing checks:

proc=point

initp

casa

proc=<schedule name><station id> vt7011wf

setupbb

ifdbb

mk6bb

auto

proc=point

fivept

onoff

azeloff=0d,0d

Send tests scan to the correlator:

From the Mark6 terminal:

gator <slots> <scanname>.vdif ~/

Wait till the prompt reappears: ~5 minutes

dqa –d <scanname>.vdif

Wait till prompt reappears ~ 5 minutes

Send files to Haystack:

scp <scanname>_*.vdif evlbi1.haystack.mit.edu:/data-st11/vgos/

Wait for confirmation from correlator that all data look good

Procedure to remove module for shipping:

mk6=group=close:<slots>

mk6=group=unmount:<slots>

Turn key OFF on disk module

mk6=mstat?all

Transfer log session file

log=station

Open terminal window

plog –l This is an “L”

vlbi session pre-checks for mark iv, vlba and dbe systems · readings of 0.1–2.0 on the...

Documents