Before today, I have written many articles that were concentrated on one topic in SQL Server or Perl or Visual Basic.
Starting today, I am going to kick off a full blown development exercise named PROJECT ALPHA. The purpose of this project is to create a test environment for developing a PERL based Extract Translate and Load (ETL) solution.
The following is a list of business requirements for this ETL project.
- A runnng trace file will be created in the LOGS directory.
- Connect to an FTP server using basic authentication.
- Download a file from the FTP server into a INBOUND directory.
- Remove the file from the FTP server.
- Import the data file into a STAGING table.
- Call a stored procedure to UPSERT the data into the production table.
- Create a directory with today’s date in the ARCHIVE directory.
- Move the data file from the INBOUND directory to today’s directory under ARCHIVE.
All these steps need to be restartable. In short, a job that fails at step 4 can be re-executed without any human intervention.
We will need to create a virtual test environment that has an File Transfer Protocol (FTP) server. The best way to allocate resources such as memory, cpu, and video is to know what hardware you have before giving it away to the virtual machine.
The is a great utility from CUPID called CPU-Z which lists this information.
A screen shot showing 8 virtual cores.
A screen shot showing 4 GB of memory.
Windows explorer can be used to figure out how much space can be used to create a virtual hard disk.
With all this information jotted down in our notepad, we now can move onto the next step, allocating a virtual hard disk from which we will create a virtual machine. Next time, we will be using Virtual Box from Oracle to perform this task.