Recently at work, had a need to rerun some load-testing numbers, but got stick since our internal servers all had builds we were looking at, or weren’t setup, or yada, yada… so we turned to EC2, with overall pretty positive results.

Some background – we build a web-based portal application that runs on a lightweight J2EE stack (Tomcat, Spring, Liferay) and do most of our testing w/jMeter. So, the process of setting up to loadteest on Amazon looked like:

• Build AMIs using Centos + our Applications (this took a bunch of hours, coming up to speed with Amazon’s tools for images – mostly the usual self-signed cert woes) — this was much easier since we have a one-click build for probably 80% of our full-machine footprint (that last 20% is a doozy though)
• Add a local LDAP server (OpenDS) to the image – was faster than troubleshooting some odd connectivity issues to our usual infrastructure
• Fire up some Amazon instances
• Grab one of the existing public AMIs w/Linux + Jmeter (Ubuntu in this case)
• Upload our scripts and start running

Net cost: ~ $40

Thoughts on the experience:

• Running jMeter inside EC2 is probably one of the things that makes this economically viable. Since our loadruns generate GBs, and GBs of traffic per minute in peak scenarious, paying per GB bandwidth can rapidly get expensive — but we were able to keep all the traffic in-cluster
• The fact that AMIs don’t by default have persistent storage (though EBS or new AWS features get around this – for more $$$) is actually a plus. We can run a test, shut down the image, bring it back up, and we have a clean image again to run to re-verify
• We were looking for stress/soak/smoke testing – so general benchmarks. For this need, we were not yet at the stage of verifying performance in a particular hardware or other environment, we just needed rough approximate numbers
• We chose not to use Amazon’s loadbalancing infrastructure, but might in the future
• EC2 actually let us run all our test scenarios (we have 3-4 main ones) in parallel, on spun-up instances – this cut our time to completion from something like 24-48 hours if we ran them serially to closer to 5hrs after setup was done – a big savings, especially for multiple runs)

All in all, it was positive experience, with some next steps we’re looking to incorporate:

1. Automate building AMIs – we did this by hand, and it was far and away the most time-expensive part (8 hours) and when we got errors early on we were immediately suspicious of configuration errors
2. Incorporate this into our Continuous Integration system (Hudson) – if construction of AMIs was automated (and Amazon’s tooling looks easy to automate) we could schedule this to run automatically, on a nightly/weekly basis – which would be nice, since right now this is a pretty heavily manual process – also, paying by the hour, the overall cost is a lot less than trying to manage with dedicated systems (and coordinate scheduling for the same)

Recently I’ve grown increasingly fond of the concepts embodied in the “Behavior Driven Development” (BDD) outgrowth of “Test Driven Development” (TDD) and “Domain Driven Design” (DDD) . BDD is a neat twist on TDD and DDD that strives to more closely tie automated tests with both the “ubiquitous language” terminology espoused in DDD, along with an Agile-style tight integration with business users in defining the TDD tests.

A great challenge with implementing TDD I think has been the difficulty in writing tests first — often driven by a difficulty in amassing sufficient detail in terms of requirements and acceptance tests to truly generate specifications for test-first development. Too often TDD ends up being a case of writing tests after-the fact to test code which we’ve written to incomplete conceptions of what the code needs to do.

A number of toolkits have emerged recently for BDD — and recent reading at http://www.agiledeveloper.com (which has some truly awesome content) led me to a PPT on BDD using easyb. Easyb provides some tools (built around Groovy) for restructuring your code to have unit/acceptance tests written in a smooth DSL — this especially appeals to me given it’s close congruence with the way my team at CampusEAI has moved to handling our requirements and user-stories.

Cutting to the chase, easyb lets you express acceptance tests in code that looks like:

These scenarios can then be directly tied to acceptance tests in code, and there’s even nice support for putting them in without implmeentations to be marked as “pending”. The overall sytax looks inspired by RSpec, but tied into Java. There’s even some nice reporting builtin to handle producing reports — plaintext here:

– looks like this would hook in well to Maven to incorporate into something like a Maven site/build process, with some tasks existing — dreaming about a report in Hudson to graph this….

Some interesting sakai-ux discussion tied in with thoughts I’ve been having recently related to native vs. integrated services and content in portal-type environments… where do the lines get drawn, and how do you handle horizontal services?

From a functionality/architecture point of view, the idea of loosly coupled integrations with external services like wikis, or Google Apps, or other tools seems very attractive.

however

There are significant horizontal capability components that while not impossible to resolve may complicate that scenario. Initial thoughts:

• TOS – ensuring users understand external, but integrated tools may have separate terms of service, SLAs, data ownership/retention policies, etc.
• import/export I think the ability to “package” and port a course/site/project or archive it is something that lots of people want. Again possible (treat external resources as links? Embed their content? Cry?) but I think it has some strong implications in user experience and architecture
• Search – this one might actually be a bit more straightforward

I will make the observation/parallel from the portal world – CampusEAI is currently heavily involved in building out a social portal, that combines the integration of enterprise services & applications with natively managed content like blogs, wikis, discussions, profile, etc. As we continue to get further down this road, some interesting intersections between user expectations and boundaries between external and internal content continue to present themselves up.

So far the balance we’ve come up with is largely – some stuff is in, some stuff is out, but there’s continual tension on the boarders of that distinction, and I’m not confident that that particular firewall will hold or be appropriate in the long run.

Not sure I have good answers for you Just a brain-dump of my internal thought processes these days.

In Dallas at the Jasig Conference, getting ready for the Board meeting where we’ll be looking at licensing, incubation, and some other strategic concerns related to open-source in HigherEd. Already had some great conversations about CAS & credential replay, Peoplesoft integration, and open-source economics & dynamics.

P.S. Ian Dolphin clarified over beers last night that he has not actually hit 2 million miles, and that’s a meme that Chuck Severance has propagated through the net space

PR-GB.com… News from origin – Hobsons Debuts Full CAS Integration at CGS Annual Meeting: “”

(Via Google.)

Another vendor server higher-ed supporting CAS out of the box can only be a good thing easing integration across the board.

ландшафт. the exhaustion that was the combined CampusEAI Portlets2008 and Annual conference is now behind us, and it seems like time for some reflections and observations. Hopefully some of these items will be items which I expand upon at a future date, but in no particular order, dumped straight from my brain:

• JSR-168 is here! Everyone really wants to write good standards compliant portlets. Architecture and engineering is a harder sell (or at least the time/cost trade-off) but there’s wide consensus that standard portlets are the way forward — at least excepting a couple of us widget fans
• SOA is something people are interested in, but that there’s been relatively little forward progress on. Some is governance. Some is tools (SOAP, WSDL, and what’s this REST thing?). Some is just that it’s big and strategic, and there’s many tactical must haves. I suspect some of it is also that much of our interesting data/services are locked in vendor platforms that have shown little interest in opening up. Though, a small trend does exist of creating SOA-style services to reach into vendor platforms and extract data from them
• Mobile wasn’t as big as I thought it would be. Not sure why. Most people seem to be interested in the abstract, but with few concrete plans. Maybe my iPhone has clouded my vision, but I do wonder if we’re going to get blindsided come fall — our target demographic is basically 18-22 year olds, afterall…
• AJAX in portlets is still hard. There are some tricks like wrapper divs, namespacing, and builtin support and integration patterns, but it’s still not a common practice.
• Identity Management is big. Governance is a big thorny issue, though many IT departments are rolling out vendor products from big players (Oracle, Sun, a little IBM) in the interim, tho ugh the exact scope of those items is somewhat unclear.
• Oracle is really putting portlets in lots of interesting places. Webcenter. Product mashups. Inside BI tools, and other GUI devices. I think they’ve probably embraced the architecture more than any other major vendor which is an interesting trend.
• Lots of awareness, and wanting to look at uPortal 3. Ooohs and ahhs over both the AJAX D&D, and maybe more importantly the new content adding UI — good going Jen!
• Really beautiful portals — some, though not all new portals really seem to be breaking out of the lots of boxes approach, or at least wrapping it in neat functionality like Boston College’s Agora design. Nifty trend. Sign of maturity?
• Community Development is hard. Aligning roadmaps, agreeing on implementation strategies, and putting all the pieces together is challenging. Even more so, justifying “doing it right” (and fit to share) versus quick and dirty, or getting a student up and running was a big trend. Makes my inner-engineer quail, but my inner-economist says that throw-away code lowers the barrier for solving problems, which is a good thing. Evolution isn’t always pretty and all that.
• Lots of desire for training, best practices, and advice on policy and governance. Real role for communities of practice, not just code and software.
• Increasing interest in “Enterprise Learning Management”. Lots of worries about migration, but the beginning of seeds wondering whether our current platforms are sufficient for a foundation for the next 10-15 years, and University strategic goals. Of course, some of this is the “enterprise IT guys” getting pulled into the LMS discussion for perhaps the first time in many places.
• Good beer is key to facilitating interesting non-session discussion. Content is king on the program, but largely only because it gets people in one place and produces interesting spontaneous interactions. Hands-on is something everyone wants, but it’s not clear a conference composed of many 1 hour sessions is the right format to deliver it.
• University IT teams wear many, many hats.
• British Universities seem to have a much richer and more abundent IT project management structure than (most) American schools. Really interesting thread about Imperial College in London blending ITIL and Agile methodologies.

Ira Fuchs from Mellon is talking about Community Source, and the larger dynamic within the community — successes, attitudes, opportunities, and risks.

zotero is interesting, both as an example of an open-source project that like Firefox has had viral adoption, with a strong community marketing and communications effort. I also wonder if there’s a convincing Sakaibrary integration possibility, perhaps a direct gateway into the Citations Manager tool that let you save or push bookmarks into the citations manager.

Vertox – a plug-in for a plug-in is interesting. Video tagging is a neat application, though in many respect I’ve always thought the combination of OCR and voice recognition to auto index text & words mentioned in a clip, and index them would be potentially more transformative than manual tagging & citing.

Clay from Georgia Tech shot me an email recently which spurred me to try and put to words how my thinking has evolved about the relationship of an enterprise portal and Sakai, and where these technologies and communities are heading.

In general I think the focus of “enterprise portals” has always been one of integration and convenience, and as a result these products are moving towards being the place that knits together all the attention streams a user might have across the digital (and non-digital) campus. I think there’s a couple key use cases, some of which have more successfully been deployed than other.

1. One stop shopping (typical) + SSO
2. Summary Views & Aggregation

Less commonly actually implemented, though often talked about/pitched: 3. Dashboards 4. Actionable Intelligence (you have overdue books, return them!) 5. Deep aggregation (e.g. pulling in all the announcements from different systems and putting them into one stream)

In addition to portals focused on horizontal integration, I think we’re starting to see vertical integration around “portals” in Learning, Collaboration, HR/Admin, SIS, Libraries? and other clumps of functionality. Some of the goals around bundling related tools together are similar, but focused around a particular toolset, or context. At some point these could probably decompose into the “lots of tools/portlets in the uber-portal” that I think represented the portal thinking years ago, but I think the reality is market forces, as well as organizational and reporting structures make that unlikely to happen any time soon.

I suspect the interrelationship w a product like Sakai to a portal is mostly as a provider of information/data — pushing out items like announcements, scheduling, files in resources, and exposing them in a different context. Ideally if we shift our thinking more along the line of wire protocols (RSS, Real SOA, RESTful APIs) this I think positions us to also start doing “network integration” where Sakai can also start talking with and working with say Banner, or Kuali FS, or Facebook, or whatever platform. I’m very impressed with CARET’s mySakai work, and think John Norman’s vision on this is similar to the kind of plan I’d outline as benevolent dictator of the Sakai universe.

Along this line, I’ve scheduled another LMS-Portal integration BOF for JA-SIG and would like to use this project as the testbed for both a WG, and an incubated integration project within JA-SIG. I think a lot of the architectural level aspects should really span LMS’s — e.g. if we do it right, ANGEL, D2L, BB, and everyone should be able to use the same protocols, though Sakai seems an ideal reference implementation. I admit to being weak on knowledge of the IMS-spec side, and am not sure whether there’s work on that front we can leverage as well. So far what I’ve seen at least on the TI front has been less API/Data centric than I think we need to go though, though Enterprise seems promising.

One particular short-term item I’d like to see Sakai expose more broadly is the group contexts expressed in the form of class enrollments & particularly ad-hoc groups represented by project site membership. In many respects I think this is the most useful data in Sakai — it’s a social-network like context that integration with and hooking other systems into seems quite valuable. Enterprise grouping systems like Grouper while promising architecturally seems to have had slow adoption, and I suspect fitting systems like Sakai with something like OpenSocial or Google Contacts-like APIs to mesh groups together may get us farther faster in the short run.

Well the questions are pouring in (mostly due to my tardiness in writing this kind of announcement) and so, without farther ado…

What Happened?

While it still feels a little strange to say it, as of 2 Fridays ago (2/8) I am no longer employed at Rutgers University. Over the last 9 years as first a student, then staff member, I’ve had the chance to: first study under, and then work with some incredible people. I’ve gotten to watch projects and services grow and evolve into solutions that are used every day by tens of thousands of students, faculty, and staff.

Before addressing my personal situation, I feel the need to speak a bit about the Rutgers Sakai deployment which up until now has occupied so much of my thoughts and energy. I was fortunate enough to see myRutgers grew into a service providing tools and services to every student at Rutgers. Sakai usage is currently somewhere on that curve, with usage growing by leaps and bounds. This Spring’s semester in many ways feels like a qualitative shift in the nature of the service — marked by a huge increase in the number of students asking “where’s my class’s Sakai site.” This semester these questions are particularly significant, as many of them are coming from students in classes where either:

1. Class was not yet in session. This is a big change from the dynamic in previous semesters where students typically visited the first meeting of their class, and were then directed to visit the Sakai site. Now students are looking to visit the Sakai site to see the syllabus, readings, and get a leg up on going to that first class.
2. Their instructor had not created a site. Sakai seems poised to make the jump into ubiquity, as in some students minds it’s already there.

Now to handle the really common question — if the Rutgers Sakai deployment is so clearly poised for greatness, where am I going and why? Well…

Starting this past monday (2/11) I have taken a position with the CampusEAI Consortium, where I will be serving as the Director of Open Source Solutions. Recent years have seen a huge upswing in the popularity, and visibility of open and community source solutions in Higher Education. Sakai, uPortal, CAS, Kuali, and othes have garnered attention, awards, and deployments. Due to significant interest expressed by member institutions, CampusEAI is looking to complement its existing strengths on the Oracle platform with broader offerings in the open-source space.

Answers to some personal-ish questions:

Are you moving to Cleveland?

No, I’m going to be based out of NJ, though Continental is certainly getting a good chunk of my time for the next few months as I schlep back and forth.

What does Lisa think?

She’s excited. Well, more excited when I’ve been gone < 2 days as opposed to > 3 days…

What do the kids think?

The kids are still getting used to not picking me up at Rutgers. They think it’s really funny that daddy works somewhere they can’t see. Sunday nights are hard. Phone calls are bittersweet. Coming back is good.

Aren’t you on the JA-SIG Board?

Yes. When my career change became definite I notified the board at the January video call. JA-SIG has always been a community of volunteers (stellar volunteers more often than not) and particular given my new employer’s willingness to continue backing my involvement in JA-SIG it was felt that there were no significant barriers to my continuing to serve in this capacity. As always, JA-SIG

So… is your Rutgers job open?

Yes. Though (see below) I’m hiring too…

What’ll I be doing?

So what does this mean in concrete terms? My personal definition is pretty simple. We’re looking to help members deploy solutions built on open source software. Given my background, Sakai, uPortal, CAS, and maybe even Kuali are obvious possibilities. I think however, that it’s a broader story than just support for deploying a few specific products. Many institutions have experienced challenges in building around open-source due to shortages in staffing or specific skill-sets. Others have successfully deployed open-source solutions, but been burned trying to deepen integration, or due to staff turnover (a problem which I should note also happens around commercial solutions). So the goal of this new unit is to make deploying solutions built on open-source:

1. Easy
2. Cost Effective
3. Low Risk
4. Sustainable
5. Did I say easy?

Basically the goal is to allow schools to leverage the strengths inherent in the open-source development model:

• Try before buy
• Rational licensing and cost-containment (instead of getting wracked with heavy licensing burdens as you get “too successful”)
• Open implementations, generally of open standards
• Economy of scale versus custom developed institution-specific software
• Freedom from vendor roadmaps and strategy shifts — even to go as far as obtain competitive bids from multiple vendors on the same solutions
• Peer interaction with really bright people working hard to solve the same problems you see

So that’s the goal. Make open-source easier, removing barriers for schools large & small — the kind of topics that have continually been commented on lists, in journals, and at conferences. Reducing installation pain. Helping with patch management. Providing support and training. Taking the pain and risk out of going open-source, all while working to make strategic contributions to enable the production of more good software.

It should be exciting.

P.S. Did I mention we’re hiring? Drop an email talking about your love for open-source, and how you really want to join in making it easier: jason_shao@campuseai.org. Oh, and mention you saw the posting in my blog

Sitting on a flight, returning from the Sakai conference — still trying to take everything in. There’ll probably be more musing on the significance of specific items coming up, but things that struck me enough to want to brain dump were:

• There was wide consensus during the planning sessions that there’s a desire to focus on quality: reliability, testing, performance, and other traits, over new feature development.
• The community is moving from a development -> production mindset. The transition of many of the core schools from pilots or development efforts into full-blown production instances has certainly changed priorities and outlooks.
• Increased desire to pick up open-standards in preference to inventing our own. JSR-168 (Unicon demoed a cool Portlet-Tool for Elluminate integration), JSR-170 and the good work Ian is doing to integrate with repositories like Jackrabbit and Xythos, CAS Authentication (vis-a-vis Dan M.’s sweet CAS-embedding UserDirectoryProvider)
• There’s a lot of commercial activity around Sakai. RSmart, Unicon, Oracle, IBM, bit players like Mark Norton & Zack Thomas. I sat in a presentation about the work I did with FIDM for CampusEAI. I even talked with another developer who has already resigned (as of Jan 1) from his university to join his part-time venture (with others) full-time. Certainly a large, vibrant marketplace.
• Many sub-groups are organizing around Sakai. In addition to our very own NYC Regional group, there are groups in California, Australia, the Netherlands, and other places. They’re holding events, sponsoring training, and moving forward.
• It’s not just regions — there are an increasing number of functionally aligned teams. Developers, Designers, and Managers are the best organized and served. I also saw a lot of User Support people or academic technologists as well. This is the group I suspect may be the next to organize — a CAFE track focused on bringing user support people up to speed or sharing experiences/resources would probably be really valuable.
• Lots of parallel activities. Many examples of SIS integrations, library integrations, documentation & training, tool development. Unfortunately, communication barriers and other difficulties seem to be producing several duplicate/parallel efforts (e.g. Yale’s SignUp tool, EDIA’s signup tool, Stanford’s efforts around this space) though there is a desire to collaborate.
• Some stuff is still too hard: authentication integration, CourseManagement integration, libraries work, documentation, training are all pain points, especially for smaller teams/schools.
• Strong community. It’s easy to get lost in a group of smart, affable people moving towards a common purpose. Had an excellent time, and there’s a good sense of camaraderie weaving throughout the community. People are friendly and helpful.
• Twitter – yeah, it’s kind of narcissistic, but at an event or convention it can certainly be a lot of fun. Both to do self-organizing (e.g. dinner?) and to pull in people who are in the circle, but not present.