User:Andr3w
EC2 Cloud Servers
These can be controlled from http://tracker.napier.ac.uk/progzoo and http://http://progzoo.net/replacelist.cgi
Andrew Cumming runs the server. Andrew's Home Page
- Andrew's Blog
- Lecture 2 Support
- Napier Students Progress
- Assessment 2008
- Assessment 2012 Flags of Brazil
How progzoo works
Progzoo relies on WikiMedia (which uses php and mysql), Apache, Perl, XSL, UML.
As a Sysop I can edit
Front end
ProgZoo uses mediawiki for the questions. The extension means that authors can include normal wiki markup plus some custom tags. The progzoo extension permits the following tags
$whoMayNotBeNamed = array();
$wgParser->setHook( "question", "question" );//Typically include default program and an answer
$wgParser->setHook( "hint", "hint" ); //User must click to see the hidden text
$wgParser->setHook( "shell", "shell" ); //Allows program 'shells' to be reused
$wgParser->setHook( "crash", "crash" ); //A pacman game
$wgParser->setHook( "soak", "soak" ); //A button to execute all programs on the page
//used to test the serverunder heavy loads
- ProgZoo extension in /home/andrew/public_html/progzoo/w/extensions
- progzoo.php - the hook, when a question tag is found it runs them through a couple of xsl files.
- question.xsl - assembles the user controls (the text box and the Run button)
- blurb.xsl - the question text is extracted and processed by wiki
- progzoo.php - the hook, when a question tag is found it runs them through a couple of xsl files.
- Zoo skin
The Zoo skin is based on Mozilla's Devmo skin. The Zoo skin has language tabs to allow users to flip between Java, C#, Perl,... It hides most of the Wiki controls. Progzoo 'should' work with other skins.
Back end
A number of servers are used to process users programs. Access to the servers is limited by the queues. Currently there are three queues, javaq, crunchie and cardington, these are represented by the structure $mList which is at the beginning of /home/andrew/public_html/progzoo/r.cgi
my $javaq= {q =>'java',
servers=>[{url=>'http://146.176.166.25/guest' ,maxProcesses=>4},
{url=>'http://146.176.166.22/guest' ,maxProcesses=>4} ] };
my $crunchie = {q =>'crunchie',
servers=>[{url=>"http://146.176.166.7/progzoo/j.aspx" ,maxProcesses=>1}]};
my $cardington = {q=>'cardington',url=>"http://146.176.166.22/guest" ,maxProcesses=>4};
my $mList = { java=>$javaq, cs=>$crunchie, vb=>$crunchie,
cpp=>$cardinton,pl=>$cardinton,py=>$cardinton,php=>$cardinton,rb=>$cardinton,
svg=>$cardington
};
Note that java is the heaviest load, two machines are attached to the javaq. The queuing algorithm will choose the machine with the fewest jobs running (it only counts processes from the queue). The first listed is prefered. Crunchie is dedicated to Windows and so handles C# and VB. The queue Cardington handles all other languages, it shares the second Java machine.
The number of processes maxProcesses allowed is 4 (number of cpu's times 2). The amount of memory dedicated to the UML is 228M. These values took some time to establish, too much memory (512M is too much) and the host falls over, too little and the virtual machine pack up when running java graphics programs (128M is too little).
The UMLs are run by user andrew from a screen session. Use of cow (copy on write) files means that the image pz.fs cannot be damaged.
rm pz.fs.cow;./kernel64-2.6.24 ubda=pz.fs.cow,pz.fs mem=228M eth0=tuntap,tap1,,192.168.0.1
UML image The UML image pz.fs includes mysql, java and all the compilers required. It runs apache. The perl script f.cgi compiles and executes the code. Before compiling and executing it uses ulimit to restrict
- the memory
- cpu time
- elapsed time
- number of processes
These are all very generous but hopefully not fatally so.
Queues
The queue is stored in /tmp/ for example /tmp/java and /tmp/crunchie See Panic for details of how to fix broken queues.
Procedure for changing the image
To change the image (if we introduce a new language or library for example) we must start the VM without COW.
#Stop apache in the host... /etc/init.d/httpd stop #Switch to the guest screen screen -R shutdown -f -h now #restart without cow ./kernel64-2.6.24 ubda=pz.fs mem=228M eth0=tuntap,tap1 #Do the update shutdown -f -h now #restart WITH cow rm pz.fs.cow screen -d -m /home/andrew/kernel64-2.6.24 ubda=/home/andrew/pz.fs.cow,/home/andrew/pz.fs mem=228M eth0=tuntap,tap1 umid=umid #Detach the screen Ctrl-A D #restart apache in the host /etc/init.d/httpd start
MediaWiki - Changes to LocalSettings.php
$wgExtraNamespaces = array(100=>'Java',110=>'Ruby',120=>'Perl',
130=>'CS',140=>'CPP',150=>'Python',160=>'VB',
170=>'C++',180=>'C#',190=>'PHP',
200=>"SVG");
$wgNamespacesWithSubpages[NS_MAIN] = true;
foreach ($wgExtraNamespaces as $k=>$v)
$wgNamespacesWithSubpages[$k] = true;
include("extensions/progzoo.php");
include("extensions/ConfirmEdit/ConfirmEdit.php");
include("extensions/ConfirmEdit/FancyCaptcha.php");
require_once( "extensions/PathFunctions.php" );
Hardware
There are whole bunch of servers dedicated (at least partially) to progzoo:
| IP | Name | OS | Features | Firewall access | Domains/Notes |
| 146.176.166.2 | iaeste | Fedora 13 | 2 Gig | 22 | Not much used |
| 146.176.166.7 | crunchie | Windows 2003 | none | SQL Server and Oracle and DB2 | |
| 146.176.166.14 | Niko | Fedora 13 | 2Gig Dual Core | 80 | web proxy |
| 146.176.166.22 | sqlzoo4 | Fedora 20 | 2 Gig | none | Gifted to Brian Davison |
| 146.176.166.43 | puglia | Fedora 20 | 2Gig 4 CPU | none | cardington - serving python for progzoo; postgres server for SQLZOO |
| 146.176.166.46 | sqlzoo5 | Fedora 20 | 1Gig Suffering | none | Unused |
| 146.176.166.55 | tracker | Fedora 20 | 80 433 | tracker.napier.ac.uk | |
| 146.176.166.58 | warp | Fedora 15 | 8 core 8 Gigs | 80 | sqlzoo.net progzoo.net csszoo.net |
| 146.176.166.140 | ManOfSteel | Fedora 20 | 8 core 8 Gigs | none | Progzoo Java Engine AND SQLZoo user MySQL |
Getting started on Fedora 18
Copy the cgi executable
scp '146.176.166.42:/var/www/cgi-bin/*' /var/www/cgi-bin/
Make space for copyFiles
mkdir /var/www/cgi-bin/copyFiles chmod go+w /var/www/cgi-bin/copyFiles
Get java
yum install java-devel
Must have perl modules:
yum install perl-JSON yum install perl-DBI yum install perl-CGI
Must turn off selinux
vi /etc/sysconfig/selinux
Get apache to startSet apache to start on boot
vi /etc/httpd/conf/httpd.conf
Find ServerName - set it to something that is also in /etc/hosts (localhost will do but that seems wrong)
systemctl start httpd
systemctl enable httpd.service
Allow apache through the firewall
firewall-cmd --add-service=http --permanent
Reboot
shutdown -r now
Add the new server to the list on progzoo.net /home/andrew/public_html/progzoo/serverlist/java-base.txt
Procedure for adding a new server
On the new server:
#As Andrew - this takes ages so start it first #Copy the progzoo image from another server [andrew@puglia ~]$ scp 146.176.166.22:pz.fs .
#Set apache to autostart, start apache [root@puglia ~]# chkconfig --level 345 httpd on [root@puglia ~]# /etc/init.d/httpd start #Install tunctl and screen [root@puglia ~]# yum install tunctl [root@puglia ~]# yum install screen [root@puglia ~]# yum install perl-JSON [root@puglia ~]# tunctl -u andrew -t tap1 #Allow port 80 in the firewall - edit /etc/sysconfig/iptables to include a line like -A INPUT -m state --state NEW -m tcp -p tcp --dport 80 -j ACCEPT [root@puglia ~]# /etc/init.d/iptables restart #Forward requests to guest to 192.168.0.2, edit /etc/http/conf/http.conf append... ProxyPass /guest http://192.168.0.2/cgi-bin/f.cgi [root@puglia ~]# /etc/init.d/httpd restart
iptables -I RH-Firewall-1-INPUT -i tap1 -j ACCEPT iptables -t nat -A POSTROUTING -s 192.168.0.0/16 -o eth0 -j MASQUERADE [root@puglia ~]# chmod og+rw /dev/net/tun [root@puglia ~]# ifconfig tap1 192.168.0.1 [root@puglia ~]# echo 1 > /proc/sys/net/ipv4/ip_forward
#As andrew - remove the cow and start in a detatched screen [andrew@puglia ~]$ rm pz.fs.cow [andrew@puglia ~]$ screen -d -m /home/andrew/kernel64-2.6.24 ubda=/home/andrew/pz.fs.cow,/home/andrew/pz.fs mem=228M eth0=tuntap,tap1 umid=umid
Make up the file system
Things that can go wrong
No network?
Do ifconfig check that eth0 is there in the guest...
eth0 Link encap:Ethernet HWaddr 2A:D2:27:57:7E:7E
inet addr:192.168.0.2 Bcast:192.168.0.255 Mask:255.255.255.0
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:19 errors:0 dropped:0 overruns:0 frame:0
TX packets:36 errors:0 dropped:0 overruns:0 carrier:0
collisions:0 txqueuelen:1000
RX bytes:2045 (1.9 KiB) TX bytes:4929 (4.8 KiB)
Interrupt:5
And in that tap1 is in the host
tap1 Link encap:Ethernet HWaddr 00:FF:21:AF:5B:12
inet addr:192.168.0.1 Bcast:192.168.0.255 Mask:255.255.255.0
inet6 addr: fe80::2ff:21ff:feaf:5b12/64 Scope:Link
UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
RX packets:68 errors:0 dropped:0 overruns:0 frame:0
TX packets:38 errors:0 dropped:42 overruns:0 carrier:0
collisions:0 txqueuelen:500
RX bytes:11816 (11.5 KiB) TX bytes:5017 (4.8 KiB)
Look at /etc/sysconfig/network-script/ifcfg-eth0 remove from BOOTPROTO dhcp
DEVICE=eth0 BOOTPROTO=none ONBOOT=yes TYPE=Ethernet IPADDR=192.168.0.2 GATEWAY=192.168.0.1 NETMASK=255.255.255.0
Turn off udev
To make it boot faster. Switch off in sysconfig, but you have to mount and copy /dev using tar first
The servers are all in B56, Merchiston Campus, Napier University (a Dell, marked server5 or sqlzoo5).
