Monthly Archives: February 2012

Oracle XE on Ubuntu using Vagrant and Puppet

Posted on by
2

Using this excellent Oracle Xe on Ubuntu Howto post on the OTN discussion forum i’ve created some configuration files to install Oracle XE on a clean ubuntu 11.10 box using Vagrant and Puppet. On my laptop this creates a new Ubuntu virtual machine running an Oracle XE database in 7 minutes.

I’ve created the following vagrant folder structure:

Vagrant oracle-xe folder structure

The rpm for Oracle XE can be manually downloaded from OTN: Oracle Database Express Edition 11g Release 2

The vagrant configuration file, Vagrantfile, looks like this:

Vagrant::Config.run do |config|

  config.vm.define :oraxe do | db1_config|
    db1_config.vm.box = "vagrant-oneiric-v3"
    db1_config.vm.host_name = "oraxe"
    db1_config.vm.forward_port 22, 41022, :adapter => 1
    db1_config.vm.network :hostonly, "33.33.33.10", :adapter => 2
    db1_config.vm.provision :puppet, :module_path => "modules", :options => "--verbose"  do |puppet|
      puppet.manifests_path = "manifests"
      puppet.manifest_file  = "site.pp"
    end
    db1_config.vm.customize [ "modifyvm", :id, "--name", "dev_env_oraxe" ,"--memory", "2048"]
	db1_config.vm.boot_mode = :gui
  end

end

The vagrant box that is used is a clean ubuntu 11.10 64bit server box.

The puppet site.pp configuration file:

node oraxe {
	include oracle::server
	include oracle::xe
}

The puppet module oracle has the following init.pp file:

class oracle::server {
  exec {  "/usr/bin/apt-get -y update":
    alias => "aptUpdate",
    timeout => 3600
  }
  package {
	"ntp":
	    ensure => installed;
	"htop":
	    ensure => installed;
	"unzip":
	    ensure => installed;
	"monit":
		ensure => installed;
	"rsyslog":
		ensure => installed;
	"curl":
		ensure => installed;
	"alien":
		ensure => installed;
	"libaio1":
		ensure => installed;
	"unixodbc":
		ensure => installed;
  }
  service {
	"ntp":
	    ensure => stopped;
	"monit":
		ensure => running;
/*		subscribe => File["/etc/monit/conf.d/monitrc"];*/
	"rsyslog":
		ensure => running;
	"procps":
		ensure => running;
  }
  exec {
		"/usr/sbin/ntpdate ntp.ubuntu.com":
			alias => "ntpdate",
			require => Service["ntp"];
  }
	file {
		"/etc/sysctl.d/60-oracle.conf" :
			source => "puppet:///modules/oracle/xe-sysctl.conf";
	}
	user { "syslog":
		ensure => present,
		groups => ["syslog","adm"];
	}
}

class oracle::xe{
	file {
		"/tmp/oracle-xe-11.2.0-1.0.x86_64.rpm.zip":,
			source => "puppet:///modules/oracle/oracle-xe-11.2.0-1.0.x86_64.rpm.zip";
		"/tmp/xe.rsp":
			source => "puppet:///modules/oracle/xe.rsp";
		"/etc/init.d/oracle-shm":
			mode => 0755,
			source => "puppet:///modules/oracle/oracle-shm";
	}
	exec {
		"unzip xe":
			alias => "unzip xe",
			command => "/usr/bin/unzip -o /tmp/oracle-xe-11.2.0-1.0.x86_64.rpm.zip",
			require => File["/tmp/oracle-xe-11.2.0-1.0.x86_64.rpm.zip"],
			cwd => "/tmp",
			user => root,
			creates => "/tmp/oracle-xe-11.2.0-1.0.x86_64.rpm";
		"alien xe":
			command => "/usr/bin/alien --to-deb --scripts /tmp/Disk1/oracle-xe-11.2.0-1.0.x86_64.rpm",
			cwd => "/tmp/Disk1",
			require => Exec["unzip xe"],
			creates => "/tmp/Disk1/oracle-xe_11.2.0-2_amd64.deb",
			user => root;
		"configure xe":
			command => "/etc/init.d/oracle-xe configure responseFile=/tmp/xe.rsp >> /tmp/xe-install.log",
			require => [Package["oracle-xe"],Exec["update-rc oracle-shm"]],
			user => root;
		"update-rc oracle-shm":
			command => "/usr/sbin/update-rc.d oracle-shm defaults 01 99",
			cwd => "/etc/init.d",
			require => File["/etc/init.d/oracle-shm"],
			user => root;
		"oracle-shm":
			command => "/etc/init.d/oracle-shm start",
			user => root,
			require => File["/etc/init.d/oracle-shm"];
	}
	package {
		"oracle-xe":
			provider => "dpkg",
			ensure => latest,
			require => Exec["alien xe"],
			source => "/tmp/Disk1/oracle-xe_11.2.0-2_amd64.deb";
	}
}

The Oracle-shm file is copied from the OTN message:

#! /bin/sh
# /etc/init.d/oracle-shm
#
#
case "$1" in
  start)
    echo "Starting script /etc/init.d/oracle-shm"
    # Run only once at system startup
    if [ -e /dev/shm/.oracle-shm ]; then
      echo "/dev/shm is already mounted, nothing to do"
    else
      rm -f /dev/shm
      mkdir /dev/shm
      mount -B /run/shm /dev/shm
      touch /dev/shm/.oracle-shm
    fi
    ;;
  stop)
    echo "Stopping script /etc/init.d/oracle-shm"
    echo "Nothing to do"
    ;;
  *)
    echo "Usage: /etc/init.d/oracle-shm {start|stop}"
    exit 1
    ;;
esac
#
### BEGIN INIT INFO
# Provides:          oracle-shm
# Required-Start:    $remote_fs $syslog
# Required-Stop:     $remote_fs $syslog
# Default-Start:     2 3 4 5
# Default-Stop:      0 1 6
# Short-Description: Bind /run/shm to /dev/shm at system startup.
# Description:       Fix to allow Oracle 11g use AMM.
### END INIT INFO

Kernel setting need to be set using a xe-sysctl.config file:

# Oracle 11g XE kernel parameters
fs.file-max=6815744
net.ipv4.ip_local_port_range=9000 65500
kernel.sem=250 32000 100 128
# kernel.shmmax=429496729
kernel.shmmax=107374183

And finally the configuration parameters for creating a new database xe.rsp:

ORACLE_LISTENER_PORT=1521
ORACLE_HTTP_PORT=8080
ORACLE_PASSWORD=manager
ORACLE_CONFIRM_PASSWORD=manager
ORACLE_DBENABLE=y

To create a clean ubuntu virtual machine with a new oracle xe installation all you need to do is:

vagrant up

Ready for use:

$ sqlplus system/manager@//33.33.33.10:1521/xe
SQL*Plus: Release 11.2.0.3.0 Production on Tue Feb 28 12:06:32 2012
Copyright (c) 1982, 2011, Oracle.  All rights reserved.
Connected to:
Oracle Database 11g Express Edition Release 11.2.0.2.0 - 64bit Production
SQL>

Self-service architecture to avoid water-scrum-fall

Posted on by
5

The cloud is enabling smaller teams to do more. IAAS enabled DevOps. Self-service infrastructure enables developers to take on operations. Developers can create and manage complete virtual-datacenters as required for their applications, without interacting with operations. This significantly increases productivity and speeds up time-to-market.

PAAS is now a big boost for NoOps. Operations is handled automatically by the platform. Developers no longer have to manage infrastructure. They can just deploy their applications. You just build, deploy and run. The result is even more productivity, and faster time-to-market.

These are all great developments. The biggest risk to project success, delivering requested functionality on time and within budget, is coordination and collaboration of large team effort. The bigger your project, the higher the risk of going over budget and over time, without actually delivering what users need.

Collaboration with operations used to be like this. Me: “i need a continuous-build server.” Them: “Why do you need to introduce something new? How big does it need to be? What software does it need to run? Does this software comply with our standards? We don’t have anybody available to manage this software right now. We first need to send somebody on a training to be able to manage this. We can have the hardware in three weeks, and it’ll take two weeks to install all your required software.” That’s a long time for something I could have installed in half a day.

Removing project dependencies, collaboration and coordination should be main priority for architects. Don’t try to design the whole system. Try to specify modules that can be created independently. The goal should be to get rid of large projects, by replacing everything with small projects, that don’t need to coordinate their efforts.

On most large agile projects you see a lot of upfront design and coordination, followed by small agile teams doing implementation. At the end of the project all the separate components are integrated, and tested.

The following image shows what I mostly encounter in larger projects working with multiple agile teams. In this example, four agile teams are working in small iterations, within a waterfall project.

Before IAAS and PAAS one (or more) of those blue teams would have been operations working on ordering and installing the infrastructure needed for your project. If one of the four agile teams has problems and delays this impacts the whole project, as testing and deployment is delayed.

When you remove dependencies and the need for coordination between teams, you may get the following. Four teams delivering value for the users independently. No need to create an overall spec, overall design or do overall testing afterwards.

This was one of the points i tried to make in my presentation for LAC2011: Scrum under Architecture. To do large scale Agile and Scrum you need to define teams and projects that can operate largely independently from each other. Every team should be able to have its own product owner, managing an independent product backlog and release calendar.

Turns out, Jan Bosch did a keynote at LAC 2011 on this type of architecture: Speed and Innovation through Architecture. He calls it composition oriented software engineering or Architectural Compositionality.

Speed and innovation is triggered by having an ecosystem of independent components. Every component team is self-directed. The product owner of each team has to determine the backlog of the project based on what he thinks the users of that component need. User can be internal or external. Big projects should not try to coordinate efforts from multiple teams. Instead they should just focus on adding a little extra on top of existing results from other teams.

In the illustration above, the blue team does three iterations to build something using the results of the other, gray, teams. The blue team could be a team implementing a mobile store application. It uses already delivered results of the product catalog team, the order entry team, the shopping-card team and the customer registration team.

By not trying to coordinate effort of all five teams in one project, but just using what has already been delivered, the project can be delivered much faster, and with less risk. At the same time each team can focus on improving their own component.

IAAS and PAAS are proving that small teams can be very productive and fast. The key element here is less coordination through self-service. Even if you’re not considering moving to the cloud with your IT, self-service and composition oriented architecture should be a priority.

Using littleproxy in Mule unit tests

Posted on by
Reply

My mule configurations files often contain proxy settings for connectors that are used to communicate to the outside world. However during development and testing I can’t use the normal proxy, since i’m mocking the outside world.

Littleproxy is a java proxy that you can use to replace the real proxy with a testing proxy. This avoids you having to install a proxy on your development machine and on the continuous integration server.

The following is part of a mule config. It defines an http connector that is configured to talk through a proxy.

<http:connector name="http.connector"/>
<http:connector name="http.connector.proxy"
                proxyHostname="localhost" proxyPort="50103"/>
<flow name="http.gateway.and.proxy">
    <http:inbound-endpoint host="localhost" port="50102"
                           path="gateway.in"
                           exchange-pattern="request-response"
                           connector-ref="http.connector">
    </http:inbound-endpoint>
    <logger message="http.gateway - #1"/>
    <http:outbound-endpoint host="localhost" port="50100"
                            path="dummy.service"
                            connector-ref="http.connector.proxy">
    </http:outbound-endpoint>
</flow>

Include little proxy in your maven pom.xml:

<dependency>
    <groupId>org.littleshoot</groupId>
    <artifactId>littleproxy</artifactId>
    <version>0.4</version>
</dependency>

Here’s the unit test. First i start the proxy (you could also use @Before or @BeforeClass), and then i run a regular mule unit test:

public void testShouldReturnHelloWorldThroughGatewayAndProxy()
    throws Exception {
    String msg = "Hi!";
    HttpProxyServer proxyServer = new DefaultHttpProxyServer(50103);
    proxyServer.start();
    /*
    * send message
    */
    MuleClient client = new MuleClient(muleContext);
    MuleMessage result = client.send(
            "http://localhost:50102/gateway.in?connector=http.connector"
            , msg, null);
    String payload = result.getPayloadAsString();
    proxyServer.stop();
    Assert.assertTrue("Did not receive HelloWorld!: " + payload, "HelloWorld!".equals(payload));
}