setup python and django with Oracle on Linux
Back in April 2008 I tried to get Django 0.96.1 working on Oracle and soon ran into some issues that clearly indicated to me it was a bit too early to invest into this framework. In fact, that experience, is what turned me heavily onto PHP/OCI8. But I kept Python in mind all this time because I really liked how structured the language was. PHP’s flexibility for WEB development is awesome, but it falls short as a scripting language when it comes to working with OS, networks and stuff we DBAs like to automate.
Few weeks ago I finally dived into python trying to adapt it as a replacement for some of the CORE shell scripts in my monitoring framework. I wanted to switch to SOA model where remote agents would talk to the management server via HTTP instead of SQL*Net. I quickly wrote the client side but then got stuck making a decision as to which language to write the server side component of SOA. Option a) was to continue using Oracle Application Server and mod_plsql, b) switch to PHP/Apache, c) adapt python.
After carefully weighting my options I decided to give python a chance since I wanted to learn what it’s capabilities were for the WEB applications. Django is the up and coming python framework, some even call it python application server. It recently went into production with v1, but it’s actually been around for over 4 years used as a private framework “by a fast-moving online-news operation“. It’s clean, easy to install and best of all it has great documentation and following.
I completed Django install in one day, which included compiling Python, cx_Oracle and mod_wsgi from source. All in all the process went fairly well but I kept wishing there was a good step by step writeup available online for getting it all setup with Oracle. There are couple of key compiler flags that make your life with python and Oracle a lot easier, but it was nowhere to be found and I learned about them by trial and error. I documented the whole process and attaching it here — hope this helps!
NOTE: This writeup assumes you have access to current Oracle Linux Support via ULN (linux.oracle.com) and your up2date was setup with ULN. It also assumes you are running 32bit Oracle Enterprise Linux or Red HAT equivalent:
[pyweb@oraemgc ~]$ cat /etc/issue Enterprise Linux Enterprise Linux AS release 4 (October Update 5) Kernel \r on an \m [pyweb@oraemgc ~]$ uname -a Linux oraemgc 2.6.9-184.108.40.206.2.ELsmp #1 SMP Wed May 2 14:59:56 PDT 2007 i686 i686 i386 GNU/Linux
Upgrade python to 2.6
First up2date tk and tcl pkgs to solve “INFO: Can’t locate Tcl/Tk libs and/or headers” error while compiling python:
up2date tk up2date tcl
Download python source code:
wget http://www.python.org/ftp/python/2.6.4/Python-2.6.4.tgz gunzip Python-2.6.4.tgz tar xvf Python-2.6.4.tar
NOTE: see problems trying to build python 2.6 as a shared library and Shared Library Search Paths for explanation of LD_RUN_PATH.
cd Python-2.6.4 LD_RUN_PATH=/usr/local/lib; export LD_RUN_PATH make clean ## shared is needed by mod_wsgi ./configure --enable-shared make make install ## FYI I ignored the following warning/error during make: ## ## Failed to find the necessary bits to build these modules: ## _sqlite3 _tkinter bsddb185 ## sunaudiodev ## To find the necessary bits, look in setup.py in detect_modules() for the module's name. ## ## BEFORE: ## [root@oraemgc Python-2.6.4]# python Python 2.3.4 (#1, Jul 27 2009, 07:32:09) [GCC 3.4.6 20060404 (Red Hat 3.4.6-11.0.1)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> [root@oraemgc Python-2.6.4]# ## AFTER UPGRADE: ## [root@oraemgc Python-2.6.4]# python Python 2.6.4 (r264:75706, Nov 24 2009, 19:31:16) [GCC 3.4.6 20060404 (Red Hat 3.4.6-8.0.1)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> [root@oraemgc Python-2.6.4]#
Install Oracle Instant client RPMs (needed for cx_Oracle)
up2date oracle-instantclient11.1-basic up2date oracle-instantclient11.1-devel up2date oracle-instantclient11.1-sqlplus
Download cx_Oracle-5.0.2.tar.gz from http://cx-oracle.sourceforge.net/ (click on “Source Code only” link once on that page and it’ll redirect you to that file).
NOTE: In order to avoid runtime errors such as “python: error while loading shared libraries: libpython2.6.so.1.0: cannot open shared object file: No such file or directory” I am setting LD_RUN_PATH during compilation of cx_Oracle library.
gunzip cx_Oracle-5.0.2.tar.gz tar xvf 10826.cx_Oracle-5.0.2.tar cd ./cx_Oracle-5.0.2 ## LD_RUN_PATH will hardwire the path to libclntsh.so.11.1 into cx_Oracle.so ## at compile time similar to what I used with Python ## export LD_RUN_PATH=/usr/lib/oracle/11.1/client/lib rm -rf build python setup.py build python setup.py install
Quick test of cx_Oracle:
export ORACLE_HOME=/usr/lib/oracle/11.1/client export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$ORACLE_HOME/lib export PATH=$ORACLE_HOME/bin:$PATH:. export TNS_ADMIN=/u01/app/oracle/admin/scripts/mon/TNSADMIN oraemgc.oracle-> sqlplus webproc/welcome@lmon SQL*Plus: Release 220.127.116.11.0 - Production on Tue Nov 24 17:44:36 2009 Copyright (c) 1982, 2008, Oracle. All rights reserved. Connected to: Oracle Database 10g Enterprise Edition Release 10.2.0.4.0 - 64bit Production With the Partitioning, OLAP, Data Mining and Real Application Testing options SQL> oraemgc.oracle-> python Python 2.6.4 (r264:75706, Nov 24 2009, 17:36:09) [GCC 3.4.6 20060404 (Red Hat 3.4.6-8.0.1)] on linux2 Type "help", "copyright", "credits" or "license" for more information. >>> >>> import cx_Oracle >>> >>> connection = cx_Oracle.connect("webproc", "welcome", "lmon") >>> >>> cursor = connection.cursor() >>> cursor.execute(""" ... select * from dual""") <__builtin__.OracleCursor on <cx_Oracle.Connection to webproc@lmon>> >>> for column_1 in cursor: ... print "Values:", column_1 ... Values: ('X',) >>>
Create user that will own django software:
useradd pyweb mkdir -p /u01/app/pyweb chown pyweb:pyweb /u01/app/pyweb ## [root@oraemgc cx_Oracle-5.0.2]# grep pyweb /etc/passwd ## pyweb:x:502:504::/u01/app/pyweb:/bin/bash ## su - pyweb vi .profile ------ add the following ------- # Source global definitions if [ -f /etc/bashrc ]; then . /etc/bashrc fi export ORACLE_HOME=/usr/lib/oracle/11.1/client export LD_LIBRARY_PATH=$LD_LIBRARY_PATH:$ORACLE_HOME/lib export PATH=$ORACLE_HOME/bin:$PATH:. export TNS_ADMIN=/u01/app/oracle/admin/scripts/mon/TNSADMIN ## create django directory ## cd mkdir -p product cd product ## download latest django code from their SVN ## svn co http://code.djangoproject.com/svn/django/trunk/
Configure django with Python. The steps below are largely based on http://docs.djangoproject.com/en/dev/topics/install/#database-installation
## verify where site pkgs are located ## cd /u01/app/pyweb/product/django-trunk [pyweb@oraemgc django-trunk]$ python -c "from distutils.sysconfig import get_python_lib; print get_python_lib()" /usr/local/lib/python2.6/site-packages ## link django with python interpreter ## and ## create a symbolic link to the file django-trunk/django/bin/django-admin.py ## ## as root ## ln -s /u01/app/pyweb/product/django-trunk/django /usr/local/lib/python2.6/site-packages/django ln -s /u01/app/pyweb/product/django-trunk/django/bin/django-admin.py /usr/local/bin ## [root@oraemgc cx_Oracle-5.0.2]# ls -l /usr/local/lib/python2.6/site-packages/ ## total 304 ## -rw-r--r-- 1 root root 898 Nov 24 17:41 cx_Oracle-5.0.2-py2.6.egg-info ## -rwxr-xr-x 1 root root 297255 Nov 24 17:41 cx_Oracle.so ## lrwxrwxrwx 1 root root 42 Nov 24 18:07 django -> /u01/app/pyweb/product/django-trunk/django ## -rw-r--r-- 1 root root 119 Nov 24 17:40 README ## [root@oraemgc cx_Oracle-5.0.2]# ## ## [root@oraemgc cx_Oracle-5.0.2]# ls -l /usr/local/bin ## total 8772 ## -rwxr-xr-x 1 root root 111 Nov 24 17:36 2to3 ## -rwxr-xr-x 1 oracle root 2854 May 13 2008 coraenv ## -rwxr-xr-x 1 oracle root 2417 May 13 2008 dbhome ## lrwxrwxrwx 1 root root 62 Nov 24 18:09 django-admin.py -> /u01/app/pyweb/product/django-trunk/django/bin/django-admin.py ## -rwxr-xr-x 1 root root 99 Nov 24 17:36 idle ## -rwxr-xr-x 1 oracle root 2947 May 13 2008 oraenv ## -rwxr-xr-x 1 root root 84 Nov 24 17:36 pydoc ## -rwxr-xr-x 2 root root 4451815 Nov 24 17:39 python ## -rwxr-xr-x 2 root root 4451815 Nov 24 17:39 python2.6 ## -rwxr-xr-x 1 root root 1424 Nov 24 17:40 python2.6-config ## lrwxrwxrwx 1 root root 16 Nov 24 17:40 python-config -> python2.6-config ## -rwxr-xr-x 1 root root 18054 Nov 24 17:36 smtpd.py ## [root@oraemgc cx_Oracle-5.0.2]# ##
NOTE: Whenever you want to update your copy of the Django source code, just run the command svn update from within the django-trunk directory. When you do this, Subversion will automatically download any changes.
based on http://code.google.com/p/modwsgi/
Note that ‘daemon’ mode of mod_wsgi is only available if Apache 2.0 or 2.2 is being used on a UNIX platform and where the Apache run time library has been compiled with support for multithreading. Generally multithreading support is standard except for on some BSD variants.
up2date httpd [root@oraemgc mod_wsgi-3.0]# /usr/sbin/httpd -version Server version: Apache/2.0.52 Server built: Nov 11 2009 19:32:25 [root@oraemgc mod_wsgi-3.0]#
Whatever version of Python is used, it must also have been compiled with
support for multithreading. To avoid a measure of memory bloat with your
Apache processes, Python should also have been compiled with shared library
support enabled. A number of Python binary packages for Linux systems are
not compiled with shared library support enabled. You should therefore
consider recompiling Python from source code with shared library support
enabled. If a shared library is not used, you will have problems trying
to use mod_wsgi on a server where mod_python is also being loaded.
FYI: I compiled python with “./configure –enable-shared” flag to support shared library and used LD_RUN_PATH=/usr/local/lib; export LD_RUN_PATH during compilation to avoid setting LD_LIBRARY_PATH during runtime (see “Upgrade python to 2.6″ step above)
If using a Linux system and you do not have the appropriate Apache “dev”
package installed, then “apxs” will not be able to be found when
“configure” is run. Also ensure that you have installed the correct “dev”
package for the version of Apache being used. For most Linux distributions,
the “dev” package for Apache 2 is “apache2-dev” where the corresponding
Apache package was “apache2″. Some systems however distinguish the “dev”
package based on which MPM is used by Apache. As such, it may also be
called “apache2-worker-dev” or “apache2-prefork-dev”. If using Apache 2,
do not mix things up and install “apache-dev” by mistake, which is the
“dev” package for Apache 1.3 called just “apache”.
[root@oraemgc mod_wsgi-3.0]# up2date --whatprovides /usr/sbin/apxs httpd-devel-2.0.52-41.ent.6.0.1.i386 up2date httpd-devel
Get mod_wsgi-3.0.tar.gz from http://code.google.com/p/modwsgi/:
wget http://modwsgi.googlecode.com/files/mod_wsgi-3.0.tar.gz gunzip mod_wsgi-3.0.tar.gz tar xvf mod_wsgi-3.0.tar cd mod_wsgi-3.0 make clean ./configure make make install
[root@oraemgc mod_wsgi-3.0]# ls -l /etc/httpd/modules/mod_wsgi.so -rwxr-xr-x 1 root root 290621 Nov 24 19:54 /etc/httpd/modules/mod_wsgi.so vi /etc/httpd/conf/httpd.conf --------- edit/add the following ----- LoadModule wsgi_module modules/mod_wsgi.so LogLevel info /etc/init.d/httpd stop /etc/init.d/httpd start
Configure mod_wsgi for Django
NOTE: based on http://docs.djangoproject.com/en/dev/howto/deployment/modwsgi/#howto-deployment-modwsgi and http://code.google.com/p/modwsgi/wiki/IntegrationWithDjango
NOTE: my project is called “evnt” you can use any name you desire just change it below:
su - pyweb mkdir -p www_top cd www_top django-admin.py startproject evnt ## this created evnt directory and the following files: ## ## [pyweb@oraemgc www_top]$ ls -lta evnt/ ## total 20 ## drwxrwxr-x 2 pyweb pyweb 4096 Nov 24 20:20 . ## drwxrwxr-x 3 pyweb pyweb 4096 Nov 24 20:20 .. ## -rw-r--r-- 1 pyweb pyweb 0 Nov 24 20:20 __init__.py ## -rwxr-xr-x 1 pyweb pyweb 546 Nov 24 20:20 manage.py ## -rw-r--r-- 1 pyweb pyweb 2818 Nov 24 20:20 settings.py ## -rw-r--r-- 1 pyweb pyweb 538 Nov 24 20:20 urls.py ## [pyweb@oraemgc www_top]$ ##
Setup custom directories to hold django/mod_wsgi configuration files
mkdir /u01/app/pyweb/www_top/evnt/apache mkdir /u01/app/pyweb/www_top/evnt/media vi /u01/app/pyweb/www_top/evnt/apache/django.wsgi -------- add the following ----- import os, sys sys.path.append('/u01/app/pyweb/www_top') sys.path.append('/u01/app/pyweb/www_top/evnt') os.environ['DJANGO_SETTINGS_MODULE'] = 'evnt.settings' import django.core.handlers.wsgi application = django.core.handlers.wsgi.WSGIHandler()
As root edit httpd.conf to configure your new django project with wsgi:
vi /etc/httpd/conf/httpd.conf ------------ add the following ---------- ## EVNT django project ## WSGIDaemonProcess oradba user=pyweb group=pyweb processes=2 threads=25 maximum-requests=10000 WSGISocketPrefix run/wsgi Alias /evnt-media/ /u01/app/pyweb/www_top/evnt/media/ <Directory /u01/app/pyweb/www_top/evnt/media> Order deny,allow Allow from all </Directory> WSGIScriptAlias /evnt /u01/app/pyweb/www_top/evnt/apache/django.wsgi <Directory /u01/app/pyweb/www_top/evnt/apache> Order deny,allow Allow from all WSGIProcessGroup oradba </Directory>
Setup TNS_ADMIN for httpd:
vi /etc/sysconfig/httpd --- add the following --- TNS_ADMIN=/u01/app/oracle/admin/scripts/mon/TNSADMIN; export TNS_ADMIN
/etc/init.d/httpd restart [root@oraemgc logs]# ps -ef | grep pyweb pyweb 939 936 0 20:40 ? 00:00:00 /usr/sbin/httpd pyweb 940 936 0 20:40 ? 00:00:00 /usr/sbin/httpd pyweb 941 936 0 20:40 ? 00:00:00 /usr/sbin/httpd root 1099 17430 0 20:40 pts/4 00:00:00 grep pyweb [root@oraemgc logs]#
Test setup by going to http://yourhostname.com/evnt which should produce the following page:
What’s next? Keep configuring your project by writing code to access Oracle, here’s a link to some examples which I am currently working on: Using the Django Framework with Oracle 11g.