Failure Analysis: 2005 6.7.3

==============================================================================================


D. Scheduler Policy failures

job_core_onexitrem-true_sched.pl
job_core_perremove-true_sched.pl
job_core_onexithold-true_sched.pl

E. Corefile Issues

job_core_corefiles_sched.run
job_core_corefiles_van.run

F. Signal Issues

job_core_killsignal_van.run

G. Run alone but not with others. Suspect timing issues.

job_core_niceuser_van.run
job_core_leaveinqueue_van
job_core_priority_van
job_core_matchlist_van.run
job_core_onexithold_van.run
job_core_holdrelease_van
job_core_perhold_van.run
job_core_rmkillsig_sched

==============================================================================================

Failure Analysis: 2005 6.6.7

==============================================================================================


D. Scheduler Policy failures

job_core_onexitrem-true_sched.pl
job_core_perremove-true_sched.pl
job_core_onexithold-true_sched.pl

E. Corefile Issues

job_core_corefiles_sched.run
job_core_corefiles_van.run

F. Signal Issues

job_core_killsignal_van.run

G. Run alone but not with others. Suspect timing issues.

job_core_niceuser_van.run
job_core_matchlist_van.run
job_core_onexithold_van.run

==============================================================================================

cmd_q_shows-hold.run*
	Description: We want to verify we can place a job on hold and
		that condor_q will show it to us with the hold option.
	Method: A job which sleeps forever is placed on hold when
		it starts to execute. We then verify correctness of 
		of condor_q -hold by parsing the output of the command.
	Expects: Hold to work, release to work and that it finds correct
		output when it parses the results of condor_q -hold.

job_condorc_ab_van.run*
	Description: This test verifies a basic condor-c job
		can be submitted to A and run on B and complete.
	Method: Two condor pools are set up with dynamic address and
		predefined locations to read the port. Then the one configured
		as the A sends the job off to the B side.
	Expects:  job completion on the A side

job_condorc_abc_van.run*
	Description: This test verifies a basic condor-c job
		can be submitted to A through B run on C and complete.
	Method: Three condor pools are set up with dynamic addresses and
		predefined locations to read the port. Then the one configured
		as the A sends the job off to the B side which send to C.
	Expects:  job completion on the A side

job_core_args_java.run*
	Description: 
	Method: 
	Expects:

job_core_args_sched.run*
	Description: Scheduler: Did specify args from submit file reach the job?
	Method: After successful completion the array or args as reported back 
		from the array maintained by the test perl module is compared against 
		the list output to std out by the job.
	Expects: job to run well and all args to match up.
	Questions: What if perl test module does not in itself cathc all the args
		up front? Weakness..... Inherited test.

job_core_args_van.run*
	Description: Vanilla: Did specify args from submit file reach the job?
	Method: After successful completion the array or args as reported back 
		from the array maintained by the test perl module is compared against 
		the list output to std out by the job.
	Expects: job to run well and all args to match up.
	Questions: What if perl test module does not in itself cathc all the args

job_core_basic_par.run*
	Description: Fundemental test oof the parallel universe.
	Method: A personal condor is set up to support a paralllel universe job
		on a preset number of machines(currently 4) and a simple sleep job
		is used.
	Expects: The job is expected to complete.

job_core_basic-security_par.run*
	Description: Fundemental test oof the parallel universe.
	Method: A personal condor is set up to support a paralllel universe job
		on a preset number of machines(currently 4) and a simple sleep job
		is used. this variation of "job_core_basic_par.run" has encryption
		enabled in the personal condor for the test.
	Expects: The job is expected to complete.

job_core_bigenv_sched.run*
	Description: 
	Method: 
	Expects:

job_core_bigenv_std.run*
	Description: 
	Method: 
	Expects:

job_core_bigenv_van.run*
	Description: 
	Method: 
	Expects:

job_core_compressfiles_std.run*
	Description: Verifies that compressed files in std universe
		are accessed and created cmpressed.
	Method:  The test verifies that the input file and output file are the
		same. The input file iz zipped up and the output file is unzip....
		both files are checked to see they match.
	Expects: input file contents = output file contents.

job_core_copytospool_van.run*
	Description: Vanilla test that spooled executable is same via md5 checksum of both files.
	Method:  Job is submitted on hold and original and spooled executable are matched
		via md5 checcksums. Submit event is the trigger to check.
	Expects: Submit event and then an abort event after we remove held job.

		up front? Weakness..... Inherited test.

job_core_coredump_std.run*
	Description: 
	Method: 
	Expects:

job_core_coredump_van.run*
	Description: 
	Method: 
	Expects:

job_core_corefiles_sched.run*
	Description: Scheduler limit of core file size - FAILS!
	Method: use coresize option in submit file and check size
	Expects: abort followed by reasonable size limited core file

job_core_corefiles_van.run*
	Description: Vanillla limit of core file size - FAILS!
	Method: use coresize option in submit file and check size
	Expects: abort followed by reasonable size limited core file

job_core_doublejeopardy_sched.run*
	Description: Scheduler: make sure condor does not generate an abort 
	AND the expected abnormal termination.
	Method: Queues two jobs(1 would work) and makes sure ONLY the 
	abnormal termination hits the log and not an abort event also.
	Job kills itself.
	Expects: Only abnormal termination.

job_core_doublejeopardy_van.run*
	Description: Vanilla: make sure condor does not generate an abort 
	AND the expected abnormal termination.
	Method: Queues two jobs(1 would work) and makes sure ONLY the 
	abnormal termination hits the log and not an abort event also.
	Job kills itself.
	Expects: Only abnormal termination.

job_core_env_java.run*
	Description: 
	Method: 
	Expects:

job_core_env_sched.run*
	Description: Scheduler: pass in environment entries and see they 
		make it through as needed.
	Method: Use the "environment" line for a submit file and dump the 
		active environment of the job to output file. tests results.
		POSSIBLE problem is we compare the submit file hash entry 
		for environment as the gospel truth...... Leave error potential 
		during the process of the submit.
	Expects: success and a 100% match up.

job_core_env_van.run*
	Description: Scheduler: pass in environment entries and see they 
		make it through as needed.
	Method: Use the "environment" line for a submit file and dump the 
		active environment of the job to output file. tests results.
		POSSIBLE problem is we compare the submit file hash entry 
		for environment as the gospel truth...... Leave error potential 
		during the process of the submit.
	Expects: success and a 100% match up.

job_core_err_java.run*
	Description: 
	Method: 
	Expects:

job_core_err_sched.run*
	Description: Does correct error file get used?
	Method: A bad perl script is the job and it is missing a " mark.
		So when the job dies it looks for a non-empty file
		with the name specified in the submit file.
	Expects: failed job with a non-zero error file

job_core_err_van.run*
	Description: Does correct error file get used?
	Method: A bad perl script is the job and it is missing a " mark.
		So when the job dies it looks for a non-empty file
		with the name specified in the submit file.
	Expects: failed job with a non-zero error file

job_core_getenv_van.run*
	Description: Vanilla job which tests getenv both on and off.
	Method:  We queue up two jobs with the same program and tell
		them via args wether the three seeded envrionmnet variables
		seeded by the test file are expected or not. If they are not expected and show up
		it fails and if they are expected and fail to show it fails.
	Expects: Both jobs are expected to pass.

job_core_hold_java.run*
	Description: 
	Method: 
	Expects:

job_core_hold_sched.run*
	Description: Test for jobs being submitted on hold.
	Method: Tell job to be on hold and checks this state
		after getting the submit event.
	Expects: only the submit state.

job_core_hold_van.run*
	Description: Test for jobs being submitted on hold.
	Method: Tell job to be on hold and checks this state
		after getting the submit event.
	Expects: only the submit state.

job_core_holdkillsig_sched.run*
	Description: Scheduler: Verify ability to control signal for holds and kill
	Method: Set hold_kill_sig in submit file and run it through a hold and then
		a remove while the job itself traps the signal.
	Expects: Goes on hold and them we remove it so we take all the action in the abort
		event. When we are done we cross check the signal number in the .out file with
		the one from the submit file.

job_core_holdrelease_sched.run*
	Description: Test state changes from running to hold to release.
	Method: We have a sleep job which when we see it execute we use condor_hold
		and when we see the hold event we issue a condor_release and when we see the release
		event we do a condor_rm.
	Expects: Events Hold/Release/Abort are what is expected.

job_core_holdrelease_van.run*
	Description: Test state changes from running to hold to release.
	Method: We have a sleep job which when we see it execute we use condor_hold
		and when we see the hold event we issue a condor_release and when we see the release
		event we do a condor_rm.
	Expects: Events Hold/Release/Abort are what is expected.

job_core_initialdir_java.run*
	Description: 
	Method: 
	Expects:

job_core_initialdir_sched.run*
	Description: We change the initialdir and assure log and stuff goes there.
	Method: Simple command is run but we set an initialdir and look for the job completing and 
		the log being there..
	Expects: Exit success and a logfile in the specified directory.

job_core_initialdir_van.run*
	Description: We change the initialdir and assure log and stuff goes there.
	Method: Simple command is run but we set an initialdir and look for the job completing and 
		the log being there..
	Expects: Exit success and a logfile in the specified directory.

job_core_input_java.run*
	Description: 
	Method: 
	Expects:

job_core_input_sched.run*
	Description: Verify the specified input file is used.
	Method:  specify an input file with known input and make those
		expected outcome and allow the standard output test
		to process the successful completion.
	Expects: Job completes and returns the predefined output.

job_core_input_van.run*
	Description: Verify the specified input file is used.
	Method:  specify an input file with known input and make those
		expected outcome and allow the standard output test
		to process the successful completion.
	Expects: Job completes and returns the predefined output.

job_core_killsignal_java.run*
	Description: 
	Method: 
	Expects:

job_core_killsignal_sched.run*
	Description: Run a job to trap the kill signal choice.
	Method:  Send a job which traps signals and places them in the output prior
		to leaving. A signal handler.
	Expects: Expects an abort.

job_core_killsignal_van.run*
	Description: Run a job to trap the kill signal choice.
	Method:  Send a job which traps signals and places them in the output prior
		to leaving. A signal handler.
	Expects: Expects an abort.

job_core_leaveinqueue_java.run*
	Description: 
	Method: 
	Expects:

job_core_leaveinqueue_sched.run*
	Description:  Verify with two built in tests that BOTH uses (T/F) impact the job queue as 
		desired. 
	Method: Run file fires off two tests, one for each case for leave_inqueue.
			leave_in_queue = (time() - QDate) < (2 ) /bin/sleep 3 false
			leave_in_queue = (time() - QDate) > (2 ) /bin/sleep 3 true
	Expects: returns from both tests to be zero.

job_core_leaveinqueue_van.run*
	Description:  Verify with two built in tests that BOTH uses (T/F) impact the job queue as 
		desired. 
	Method: Run file fires off two tests, one for each case for leave_inqueue.
			leave_in_queue = (time() - QDate) < (2 ) /bin/sleep 3 false
			leave_in_queue = (time() - QDate) > (2 ) /bin/sleep 3 true
	Expects: returns from both tests to be zero.

job_core_macros-dollardollar_van.run*
	Description: Test $$macros
	Method: Pass $$memory, $$mchine and a nonsense macro via both
		argument passing and environment passing. A simple perl script
		dumps the args and environment values to stdout. The test then
		examines the stdout file for the test.
	Expects: To find the $$macros.

job_core_macros_java.run*
	Description: 
	Method: 
	Expects:

job_core_macros_sched.run*
	Description: a number of macros get passed in as args.
	Method:  a number or args get sent back to the output file and
		a few are looked for in an array we load up from the output file.
		Could be more robust in time and with more understanding.
	Expects: successful completions.

job_core_matchlist_van.run*
	Description: Vanilla test that match_list_length is storing and rotating matches
	Method:  a simple job runs and completes but is left in the queue to run again
		via on_exit_remove. Each run checks that the current LastMatchName0 matches
		the MATCH_name in the current ad and if LastMatchName1 exists, it matches
		the last(stored) LastMatchName0.
	Expects: Expects an abort because we limit the job to 5 runs.

job_core_max-running_local.run
	Description:
	Method:
	Expects:

job_core_niceuser_van.run*
	Description: Vanilla test for nice_user being honored.
	Method: We queue up two jobs which expect the output file to be moved to
		them for adding to. If job which write 2 is correctly delayed 
		by nice user, then the data should be there from both jobs. Unsure about
		jobs in a big pool with resources for both jobs. The test will probably
		fail as the jobs running at the same time will last run wipe the output
		file as it is not designed for both to run at once. Maybe I should use my generic
		server and client and a shared data file? Or perhaps I should scrap the log file?
	Expects: Jobs to not run at the same timee.

job_core_niceuser_van.run*
	Description: Check that nice_user is honored.
	Method: Predetermined output in run file is checked against the order the jobs ran and 
		their output. We expect order of jobs to be 1(not nice_user) then 2(nice_user).
	Expects: data file created during the run to match data in run file.

job_core_onexithold_java.run*
	Description: 
	Method: 
	Expects:

job_core_onexithold_sched.run*
	Description: Verify on_exit_hold works with both a true and a false trigger
	Method: Run two tests and check correct behaviour.
		on_exit_hold = (time() - QDate) < (2 ) /bin/sleep 3 false
		on_exit_hold = (time() - QDate) < (2 * 60) /bin/sleep 3 true
	Expects: false job should complete trivially and true should be held

job_core_onexithold_van.run*
	Description: Verify on_exit_hold works with both a true and a false trigger
	Method: Run two tests and check correct behaviour.
		on_exit_hold = (time() - QDate) < (2 ) /bin/sleep 3 false
		on_exit_hold = (time() - QDate) < (2 * 60) /bin/sleep 3 true
	Expects: false job should complete trivially and true should be held

job_core_onexitrem_java.run*
	Description: 
	Method: 
	Expects:

job_core_onexitrem_sched.run*
	Description: Verify true and false triggers of on_exit_remove
	Method:  run two tests and verify both
		on_exit_remove = (time() - QDate) > 60 /bin/sleep 3 true
		on_exit_remove = (time() - QDate) > (2 ) /bin/sleep 3 false
	Expects: requeue on true and trivial completion on false

job_core_onexitrem_van.run*
	Description:  Verify true and false triggers of on_exit_remove
	Method:  run two tests and verify both
		on_exit_remove = (time() - QDate) > 60 /bin/sleep 3 true
		on_exit_remove = (time() - QDate) > (2 ) /bin/sleep 3 false
	Expects: requeue on true and trivial completion on false

job_core_output_java.run*
	Description: 
	Method: 
	Expects:

job_core_output_sched.run*
	Description:  Verify the specified input file is used
	Method:  Input is rerouted to output and standard expect data is used to verify output
	Expects: Trivial success with predefined output.

job_core_output_van.run*
	Description: Verify the specified input file is used
	Method:  Input is rerouted to output and standard expect data is used to verify output
	Expects: Trivial success with predefined output.

job_core_perhold_java.run*
	Description: 
	Method: 
	Expects:

job_core_perhold_sched.run*
	Description: verify true and false triggers for periodic hold
	Method: run 2 tests for true evaluation and false with true ending
		up by having the job placed on hold.
		periodic_hold = (time() - QDate) > 5 /bin/sleep 40 true
		periodic_hold = (time() - QDate) < 0 /bin/sleep 4 false
	Expects: true should pass from executed to hold to abort when we remove it
		while false should have trivial completion.

job_core_perhold_van.run*
	Description: verify true and false triggers for periodic hold
	Method: run 2 tests for true evaluation and false with true ending
		up by having the job placed on hold.
		periodic_hold = (time() - QDate) > 5 /bin/sleep 40 true
		periodic_hold = (time() - QDate) < 0 /bin/sleep 4 false
	Expects: true should pass from executed to hold to abort when we remove it
		while false should have trivial completion.

job_core_perrelease_java.run*
	Description: 
	Method: 
	Expects:

job_core_perrelease_sched.run*
	Description: Verify true/false policies for periodic release do the right thing.
	Method: run 2 tests for periodic release where one evaluates to true and the other false.
		periodic_release = (time() - QDate) > 3 /bin/sleep 6 true
		periodic_release = (time() - QDate) < 0 /bin/sleep 10 false
	Expects: True version should see execute/release/reschedule. False should see a timeout.

job_core_perrelease_van.run*
	Description: Verify true/false policies for periodic release do the right thing.
	Method: run 2 tests for periodic release where one evaluates to true and the other false.
		periodic_release = (time() - QDate) > 3 /bin/sleep 6 true
		periodic_release = (time() - QDate) < 0 /bin/sleep 10 false
	Expects: True version should see execute/release/reschedule. False should see a timeout.

job_core_perremove_java.run*
	Description: 
	Method: 
	Expects:

job_core_perremove_sched.run*
	Description:  Verify true/false policy settings for preiodic remove.
	Method: Run two tests where one evaluates to true and the other to false.
		periodic_remove = (time() - QDate) > 2 /bin/sleep 3 true
		periodic_remove = (time() - QDate) < (0 ) /bin/sleep 3 false
	Expects: False expects trivail success. True expects an executed followed by an abort.

job_core_perremove_van.run*
	Description:  Verify true/false policy settings for preiodic remove.
	Method: Run two tests where one evaluates to true and the other to false.
		periodic_remove = (time() - QDate) > 2 /bin/sleep 3 true
		periodic_remove = (time() - QDate) < (0 ) /bin/sleep 3 false
	Expects: False expects trivail success. True expects an executed followed by an abort.

job_core_plus_java.run*
	Description: 
	Method: 
	Expects:

job_core_plus_sched.run*
	Description:  Verify ability to place things in job ads using +.
	Method: +items are placed in submit file and then the job ads are parsed and verified..
	Expects: Expects to find created job ads after a successful completioon.

job_core_plus_van.run*
	Description:  Verify ability to place things in job ads using +.
	Method: +items are placed in submit file and then the job ads are parsed and verified..
	Expects: Expects to find created job ads after a successful completioon.

job_core_priority_java.run*
	Description: 
	Method: 
	Expects:

job_core_priority_sched.run*
	Description: Check that priority is honored.
	Method: Predetermined output in run file is checked against the order the jobs ran and 
		their output
	Expects: data file created during the run to match data in run file.

job_core_priority_van.run*
	Description: Check that priority is honored.
	Method: Predetermined output in run file is checked against the order the jobs ran and 
		their output
	Expects: data file created during the run to match data in run file.

job_core_queue_sched.run*
	Description: Verify queue command.
	Method: fires up a server and each job says hello to server and then they are tallied.
	Expects: 40 hellos in server log.

job_core_remote-initialdir_van.run*
	Description: Verify we can set remote-initialdir correctly.
	Method: job does a ls to determine the inode for the current
		directory and for /tmp. If they match /tmp is placed in
		.out file for test. If not, some other directory is placed
		in the .out file.
	Expects: /tmp in the .out file.

job_core_rmkillsig_sched.run*
	Description:  Verify signal used by condor_rm by trapping it and sending to output file
	Method:  trap signal and write to file
	Expects: signal number in output file

job_core_sh-loop_van.run*
	Description: 
	Method: 
	Expects:

job_core_signal-suicide_sched.run*
	Description: 
	Method: 
	Expects:

job_core_signal-suicide_van.run*
	Description: 
	Method: 
	Expects:

job_dagman_basic.run*
	Description: Basic one node dag
	Method: Single node with a pre and post script
	Expects: submitted,executing and success

job_dagman_comlog.run*
	Description: Run a dag with multiple nodes sharing a common log
	Method: Shared log by a common submit file for each node
	Expects: submitted,executing and success

job_dagman_fullremove.run*
	Description: Ensure a removed dag is completely removed
	Method: When the job starts executing we remove it and then check the log for complete removal
	Expects: expects submitted, executing and abort

job_dagman_maxpostscripts.run*
	Description: Check that throttling by post scripts works
	Method: a dedicated server is started and each node says hi and goodbye. After the run we tally 
		number running at any one time and if max is exceeded job fails.
	Expects: to see submitted and success

job_dagman_maxprescripts.run*
	Description: Check that throttling by pre scripts works
	Method: a dedicated server is started and each node says hi and goodbye. After the run we tally 
		number running at any one time and if max is exceeded job fails.
	Expects: to see submitted and success

job_dagman_prepost.run*
	Description: Run a dag and verify each nodes pre and post scripts ran
	Method:  A central server collects messages from all the pre and post scripts 
		and after it runs the log is tallied for an exact pre and an exact post 
		script count.
	Expects: job should be submitted, execute and complete successfully

job_dagman_retry.run*
	Description: Make sure a retry dag runs correctly.
	Method: A bad node return is initially passed to the post script and then the retry
		dag is changed to ignore the return so it runs successfully to completion.
	Expects: Expects to see termination from three bad tries and then expects rescue dag to run
		to completion successfully.

job_dagman_throttle.run*
	Description: Test dagman's ability to throttle number of jobs at once.
	Method: Use the throttle option and scrap the user log to see it happened.
	Expects: Thee dag is expected to run successfully and then success depends on
		good parsing of the user log.

job_dagman_splice-A.run

testA:
    Test interpolating of the $(JOB) variable in to the right hand side of
    a VARS variable for the job's submit file.

testB:
    Test inclusion of multiple splices in one dag file (which has
    jobs and parent/child relationships between the jobs) with no
    PARENT/CHILD relationships between the splices and anything else. The
    result should be each splice is treated as a disjoint graph in
    the entire dag.

testC:
    Test using a splice as a child of a regular JOB.

testD:
    Test using a splice as a parent of a regular JOB.

testE:
    Test using a JOB as a child and parent of a splice.

testF:
    Test using a splice as a child and parent of a regular JOB.

testG:
    Test using a splice as a parent of another splice.
    This also tests using a splice as a child of a splice.

testH:
    Splices which include themselves and have no parent/child relationships
    between them should all appear as disjoint graphs in the entire dag.

testI:
    If a splice has disjoint graphs, then make sure all the initial and
    final nodes are treated properly when the splice is used as a parent/child
    in a dependancy.

testJ:
    Run condor_submit_dag with multiple dag files each bringing
    in a splice.  Each main dag file (after splicing is resolved)
    should be a disjoint graph to the other dag files. This tests to see
    if the jobnames and splice names had been "munged" correctly.

testK:
    This was a dag geometry I stumbled across which flushed out a bug in
    the prefixing of nodes and the reconstruction of the node name
    index hash table. It had to do with the specific splice names
    (S2 being used legally twice) and the bug in my code.

job_dagman_splice-K.run

job_dagman_uncomlog.run*
	Description: Test Dagmans ability to have sepaarate job log files
	Method: Specify all different logs files for nodes and see that Dagman
		can follow the progess of the job.
	Expects: Job should finish successfully.

job_dagman_unlessexit.run*
	Description: Test that the retries can be stopped with particular exit values from node.
	Method: Node should not be retried as 3 is looked for and should stop the retry. We
		look at user log to verify this.
	Expects: successful completion

job_dagman_xmlcomlog.run*
	Description: Set up the dag to have a common log file but make format be xml
	Method: Runs a 9 node dag and looks to have it just work.
	Expects: successful completion

job_dagman_xmluncomlog.run*
	Description: Set up the dag to have separate log files but make format be xml
	Method: Runs a 4 node dag and looks to have it just work.
	Expects: successful completion

job_filexfer_base-input1_van.run*
	Description: Vanilla: Tests input file does not make it with filetransfer off
	Method: Specify a non local input file and leave file transfer off.
	Expects: "failok" engages negative logic. Job is successful ONLY because it fails.

job_filexfer_base-input4_van.run*
	Description: Vanilla: Verifies submit fails with input list and with filetransfer 
		off
	Method: Specifies 3 non local input files and leaves file transfer off.
	Expects: Submit error

job_filexfer_base_van.run*
	Description: Vanilla: Is executable moved correctly to initialdir/sandbox?
	Method: No explicite file transfer but executable moves and runs.
	Expects: Expects success by default moving of executable

job_filexfer_basic_sched.run*
	Description: Scheduler exercise of file transfer via cp in to out.
	Method: /bin/cp copies input file to new name explicitely asked for
		as output file.
	Expects: Expects line by line comparison of files should match.

job_filexfer_basic_van.run*
	Description: Vanilla: send a file and get it back
	Method: /bin/cp copies input file to new name explicitely asked for
		as output file.
	Expects: Expects line by line comparison of files should match.

job_filexfer_input-onegone_van.run*
	Description: Vanilla: test for failure when specified input file is missing
	Method: Specify 3 input files and have one missing
	Expects: there should be a submit error caught

job_filexfer_limitback_van.run*
	Description: Vanilla: ask for exactly 2 files returned
	Method: Asks for two of the three files produced as explicite output
		files and makes sure they arrive
	Expects: 2 files to arrive - does not test if third file did too.

job_filexfer_md5_java.run*
	Description: 
	Method: 
	Expects:

job_filexfer_md5_van.run*
	Description: create and move 200 megs of MD5 checksumed data 
	Method: Moce the data and returnit as a different name and verify same md5 
		checksum on the data.
	Expects: checksum of new file to match original checksum

job_filexfer_minus1_van.run*
	Description: Vanilla: extra specified output file creates shadow 
		exception to rerun to produce missing output
	Method: Submit file explicitly ask for 4 file but job produces only 3.
	Expects: shadow exception

job_filexfer_output-withvacate_van.run*
	Description: Vanilla: 3 files created before vacate - 
		verify their retrun
	Method: Test job produces 3 files right away and then sleeps
		20 seconds and then produces 3 more files. We vacate the job
		when we see it start to run and expect an abort from the vacate_job.
		RELIES on 6.7.x based condor_vacate_job!
		Fails because 3 files only come back when specified as output files
	Expects: Execution -> Eviction -> Abort....... and files back without asking for them

job_filexfer_output_van.run*
	Description: Vanilla: Are 6 output files returned?
	Method: Remote transfer job produces process id labeled output files
	Expects: job to run ok and have all 6 files returned

job_filexfer_sandbox-empty_van.run*
	Description: Vanilla test for removal of sandbox.
	Method: A small perl script sets up some subdirectories with unusual
		permissions and reports back via the .out file the location
		of the local execute directory established. A sleep/check loop
		watches for the sandbox to be gone.
	Expects: Successful execution of the job and then expects sandbox 
		to vanish before its eyes.

job_filexfer_streamout_van.run*
job_filexfer_streamerr_van.run*
	Description: Vanilla test for streaming of error.
	Method: A job is fired which is supposed to stream. A job is run which
			determines if we are testing stdout or stderr. It currently writes
			out 100 lines to whichever place and then sleeps 5 minutes. The
			test allows currently 1 minute after the execute state for the
			file to show up knowing that the job is still running. It has a 2
			minute timeout currently. After one minute we look for the file.
	Expects: Master scripts expects file to be found.

job_filexfer_trans-excut-true_van.run*
	Description: Vanilla: explicitely set transfer_executable... test complains
	Method: Explicitely set transfer_executable to true and specify an input file.
	Expects: Submit error expected

job_filexfer_trans-nodflts_van.run*
	Description: Vanilla: all transfers explicitely off - complains...
	Method: all of the transfer_ options are turned off
	Expects: expects a shadow exception because the job never gets transfered

job_filexfer_whento-base_van.run*
	Description: Vanilla: Inputless test of a job running with transfer files enabled.
	Method: File transfer enabled and job runs no input....
	Expects: Simply expects job to run ok

job_filexfer_whento-simpleinput_van.run*
	Description: Vanilla: job runs with input file requested
	Method: File transfer enable and a single file is requested in submit file. 
	Expects: job to run without error. The transfer file job knows what to look for.

job_filexfer_whento-withinput_van.run*
	Description:  Vanilla: Multiple input files arrive ok
	Method: multiple files are specified as input
	Expects: job to run without error. The transfer file job knows what to look for.

job_filexfer_withvacate-nofiles_van.run*
	Description: Three files are created before vacate. Verify their non-return
	Method: Have an ON_EXIT timing for file returns and then vacate job after three 
		files are created. Specifying them as output files brings them back anyways!
		so for now we fail this test!
	Expects: files to not come back

job_had_basic.run
	Description:
	Method:
	Expects:

job_schedd_job-tracking.run*
	Description: Set loose a flock of jobs and make sure all get 
		accounted for.
	Method: We queue up 400 jobs and count them as they complete 
		with a timer running from when the last of the 400 has
		been submitted.
	Expects: 400 jobs should complete successfully.

lib_chirpio_van.run*
	Description: A program linked into the chirp library is run to
		test a series of system calls.
	Method: The chirp library is used to write and read a file with
		particular data written to verifiable positions.
	Expects: to find things where left behind.....

lib_procapi_pidtracking-snapshot.run*
	Description: Run a job which forks children which fork children
		and check for proper clean up of jobs. This version relies on the
		job running long enough for the starter to get a "snapshot".
	Method: Child processes( 2 generations ) are created and documented
		by pids. After the job completes, the test looks for all of those
		processes being properly cleaned up. These child processes run
		forever if not cleaned up. An all inclusive PS is processed
		for processes still running. The job waits around 60 seconds to
		allow a snapshot to be captured.
	Expects: The list of processes are all cleaned up.

lib_procapi_pidtracking-byenv.run*
	Description: Run a job which forks children which fork children
		and check for proper clean up of jobs. This version relies on 
		environment tracking of child processes".
	Method: Child processes( 2 generations ) are created and documented
		by pids. After the job completes, the test looks for all of those
		processes being properly cleaned up. These child processes run
		forever if not cleaned up. An all inclusive PS is processed
		for processes still running. The job waits around 0 seconds to
		check for removal of child processes prior to a "snapshot".
	Expects: The list of processes are all cleaned up.

lib_userlog.run*
	Description: Our userlog library is being tested in a read/write sequence.
	Method: A program linked to our userlog library write a predefined series
		of userlog events. Then a job is run which reads this logfile.
	Expects: Compares the result of the user log read to match a predefined
		output stored in the test.

perf_busy_lynn_100_100_500_sched.run
	Description:  Generated test which does a load test in the scheduler 
		universe which begins with 100 cpu loading jobs and runs another
		100 per iteration and stops after doing 500 jobs.
	Method: Generated tests checked in as sample
	Expects: 0 for success 1 for failure

perf_jobs_sue_10_10_30_van.run
	Description:  Generated test which does a jobs test in the vanilla 
		universe which begins with 10 jobs and runs another
		10 per iteration and stops after doing 30 jobs.
	Method: Generated tests checked in as sample
	Expects: 0 for success 1 for failure

perf_xfer_deb_10_10_30_van.run
	Description:  Generated test which does a transfer test in the scheduler 
		universe which begins with 10 transfer jobs and runs another
		10 per iteration and stops after doing 30 jobs.
	Method: Generated tests checked in as sample
	Expects: 0 for success 1 for failure

soap_job_tests.run
	Description:
	Method:
	Expects:

job_rsc_all-syscalls_std.run*
job_rsc_f-direct-write_std.run*
job_rsc_fcntl_std.run*
job_rsc_fgets_std.run*
job_rsc_fread_std.run*
job_rsc_fstream_std.run*
job_rsc_ftell_std.run*
job_rsc_getdirentries_std.run*
job_rsc_hello_std.run*
job_rsc_open-N-serial_std.run*
job_rsc_stat_std.run*
job_rsc_truncate_std.run*
job_rsc_zero-calloc_std.run*
