Best thing is that the bloody annoying “Recommendations” fly out menu on docs.sun.com has died with the migration \o/

I did watch most of the Oracle presentations yesterday, as well as following Apples iPad launch. Both left me a bit “bleh”.

The iPad is shiny, and I’m sure once I’ve had a play with one I’ll decide I need one. But, it’s not quite there IMO. $500 is tempting price but why so much extra for 3G? Also, I already have a data contract on my iPhone – why can’t I use that? Bluetooth the ipad to my phone, share the data and I’ll be really tempted (I’d be willing to pay a bit extra a month to cover extra data usage). Anyway, time will tell if Apple have again managed to come up with something market changing – I’m not convinced (yet ).

Oracle and Sun was also a bit of a non event for the bits I’m really interested in. Talk of the T processors was good – they aren’t being scrapped then. Very little talk however of the sort of servers we might buy – T5220 type boxes. Will they still be developed? Will the new coolthread CPUs be affordable to us? Nothing much was said.

The suggestion that support will be simplified is also rather worrying. While Sun support can be a pain in the arse with all the options (they are currently trying to work out if we are supported for DSEE7 under JES for example), it has been affordable to us at the lower levels (generally, Silver support). If Oracle really want to simplify it to one, easy to understand level it’s going to be gold equiv at least. Can’t see it not going up massively – probably to a point where we just can’t justify it

The other big worry is how Oracle will merge the two middleware stacks. Glassfish is going to stick around, but some of the other components it was less clear. Given we are about to deploy DSEE and OpenSSO Enterprise I was hoping for some more reassurance that they weren’t going to be scrapped. DSEE will stick around I’m sure – but OpenSSO? Less clear…and even if it does, will we be able to afford to renew the contract in 18 months time?…

Glad to hear that the rumours of massive cuts to Sun staff (well, the ones who are still left working there!) didn’t prove to be true. Quite the opposite it seems – Oracle are employing more. This is a good thing(tm) IMO.

Still, the next few months could be interesting to see how it all pans out.

Anyway, happy now Tony?

Could be an interesting day tomorrow firstly, Apple event – *finally* the Apple tablet will make an appearance? Sounds from the rumours it’s going to be more iphone than mac though so I’m less interested (well, a bit less ). Definitely interested in iPhone OS4 and any new phones given I’ve just dropped out of contract…

Secondly, and probably more importantly, the Oracle Sun webcast. After the EU finally gave the go ahead for the takeover (seems to have been going on for *ages*) it’ll be interesting to hear what Oracle really plan to do with Sun.

Dropping mid range storage seems obvious. Sun are not all that strong in that area – better alternatives are around. Similarly, tape is probably out IMO (but openstorage kit more likely to hang around?). Solaris isn’t likely to be going I’d imagine – dropping that makes no sense if oracle are serious about selling the entire stack. Sparc I’m less convinced on. The T series boxes are neat in many ways, and crap in others. The high end FujitsuSun boxes with the “real” Sparcs in are just too expensive for us. I know Larry has suggested that Sparc isn’t going away but I can’t see how they can justify designing their own CPUs when there are options like the nehalams available. So, high end stays with Fujitsu, lower end becomes x86 only? Would be a shame…

The software stack is rather more murky. Java won’t be going anywhere, that’s for sure but there is a lot of overlap in the middleware – Appservers, identity/access management and to some extent, the comms stack have massive overlaps that will have to be cut. Be interesting to see what gets slashed from this lot. Virtualbox feels rather out of place – and isn’t ever going to be VMware. global desktop somewhat safer though I’d imagine given Oracle are massive users of this. Given how open Oracle are (ha!) it’ll be interesting to see how they view the opensource everything model.

We are just about to roll out OpenSSO on Glassfish with DSEE all running on a pair of coolthreads boxes so I fully expect oracle to announce they are scrapping all of that lot. Still, if only we had a nice new shiny Oracle Appserv license…

Oracle have already said MySQL isn’t going away (and was largely the cause of the EU delays) but it will be interesting to see how/where they market this. Still, some level of marketing will be novel – sometimes it’s just too much like hard work trying to get Sun to actually sell you something! No wonder they don’t make money.

In terms of staff cuts, there has to be some. Rumours of 50% seems rather drastic (but then, something drastic has to happen – Sun isn’t working as it stands). Hope friends and contacts at Sun are ok. Sun support has always been some of the best. Hope that isn’t cut too much – marketing and sales should IMO be feeling a little nervous though :-/

Oh well, should know a bit more this time tomorrow eve…I suspect I’ll get around to posting something around July

Bye bye Sun?

A couple of months ago we upgraded our Window 2000 domain to Windows 2003. We tested idsync in a dev environment and thought it was all working fine. However, since then, we’ve been getting reports of users changing their passwords on windows but LDAP only accepting their old passwords. The idsync logs showed that it all appeared to be working perfectly, the windows logs showed, well, showed bugger all really (don’t they always?). Finally today while a few of us were chatting over a coffee the penny dropped…

As AD doesn’t make passwords available (or hashes of) the LDAP server has no way of obtaining the passwords directly. What it has to do is to monitor the AD changelog (point at PDC emulator role – odd things happen otherwise with replication delays in AD) to detect when a users password changes. When it does, it flags the user as having a password that needs syncing by setting the attribute dspswvalidate to true. Then, when the user next binds to the ldap server (via mod_auth_ldap in apache, UWC, msging express, whatever) the idsync plugin to DS takes the password that it has been given and attempts a bind against the AD domain controller. If it works it knows the password is correct and squirrels it away in LDAP (at least, a hash of it) and allows the bind to succeed. Clever, and reliable until Windows 2003 SP1 when for some reason I can’t fathom Microsoft changed the behavior. Now, for 1 hour after changing a password in AD a user can still use their old password for 1 hour. Sounds a trivial change but if you have users who have their (old) password stored in email clients etc then it’s a race to see which password is synced first. If the user happens to use their *new* AD password then idsync sends that to the DC and gets it validated, stuffs it into LDAP and huzzah, the world is sweet. If however the user (or more likely, a client they have running with a saved password) uses their old password first then idsync again sends this to the AD DC which since SP1 now returns a success for the old password! Of course, idsync the stuffs that into ldap and never bothers syncing it again – why should it, it has the new password doesn’t it?

Arggggghhhh

Of course, it’s not all Microsoft being shit. We have had a call logged with sun for this for several weeks now (case ID 37917017 if anyone reading can look it up). They haven’t managed to come up with a reason why this is happening despite in v6 of DS it’s actually a documented issue on suns site (see http://docs.sun.com/app/docs/doc/820-2487/isw-bugs-known ). For version 2005Q4 Sp1 which is the version that supports 2003 it isn’t mentioned at all :-/ I’ll suggest a doc update tomorrow when I get into work. In the meantime, hopefully I’ve scattered enough of the relevant keywords through this post that others suffering the same odd behavior will find it.

BTW, the fix it alledgedly described in http://support.microsoft.com/?kbid=906305 although we’ve yet to confirm this – fiddling with the registry on live domain controllers during signup week isn’t something we can just do without thinking carefully… Also, that doc seems to suggest that it’s NTLM auth that is effected – it’s certainly LDAP and we’ve also confirmed it’s an issue with radius auth (windows desktops work properly and reject the old password immediately – convenient).

Finally, is it just me who finds the end of that document rather laughable?

Note This behavior does not cause a security weakness. As long as only one user knows both passwords, the user is still securely authenticated by using either password.

If a user’s password is known to be compromised, the administrator should reset the password for that user. The administrator should ask the user to change the password at the next logon to invalidate the old password as soon as possible.

Ask the user to change their password as soon as possible when you know it’s been compromised???? ARGGGHHHHHHH

Today it seems her replacement has been announced (or possibly leaked, not sure there has been an official announcement yet). Apparantly Kim Jones was (is) vice president of global education – could be interesting. Maybe Sun will decide if they are serious about getting into education or if they have given in to Microsoft.

In other Sun related news, seems that Sun have now come back to us with a much more sensible price for JES. I know some of the people involved in sorting this read this. Thankyou – J, and N I owe you a pint

Also, don’t try to use truss -f to follow the java process that is doing the syncing – holy crap, what a lot of output. No wonder Java chugs a bit at times.

Still, in other news it seems Sun do want us to stay – they have (finally) come back with a more realistic price for JES. Now, if only they would make it work as well.

All sounds great – a few years late possibly but it all sounds good.

Too late for us? Quite possibly.
Too late for Sun to take on Exchange? Maybe. It’ll be interesting to watch

Our initial contract with Sun expires this summer. 3 years on it nearly works (which believe me, is progress on what we had to start with) and time comes to renew our contract. Two things seem to have happened. 1) we are being offered no discount which is a fair hike on what we paid to start with (admittedly, it was a great deal – or would have been if it had worked) and b) Messaging server and Calendar server (both of which we rely on) have been pulled from the JES suite and put into their own product…which is an extra charge on any new contract.

Upshot of this is that we are being asked to pay a 600% increase in yearly licence and support costs. This is to license the product that doesn’t quite work yet and for support that seems to involve posting our question on public forums… Hardly a bargain.

Last I heard was that they might be able to do something and may just about be able to get it down to a 300% increase. Well, if thats the best they can do then sod them, we’ll be moving to exchange. Actually, given the timescales it’s not really realistic to move this year so I suspect we will have to pay up for another year while we plan the migration (it also seems that the JES contract comes with 60hrs of PS time and $10k worth of training…I’ll be demanding a sun PS guy to come sit in the office for 60 hours while I throw things at him if that’s the case. Maybe they could help us build Exchange?).

Of course, once we move to Exchange it’ll intergrate nicely with Sharepoint. Outlook will finally work properly (you know, bloaty emails, winmail.dat and all the other shite that “power users” seem to require). Hell, it’ll even work with Outlook 2007 (come on Sun, I don’t *care* how complex it is – even you can’t claim that 2007 crept up on you as a surprise).

Once we have Exchange and Sharepoint then IIS becomes more appealing. Given how Microsoft aren’t known for their openness this can only be a one way move. Sun, are you listening? Prove me wrong and give us a sensible price for the software (which is barely past Beta if you are honest – as several of your own engineers have told me). One of your Java Campuses is a gnats prick from becoming largely a Microsoft shop

JES doesn’t work reliably. Support is patchy (improving I admit) and if I could be bothered to go through all of our case history I’m sure I’d find at least 85% of our calls have resulted in the problem being due to a bug in Suns software. Why should we pay a small fortune to debug *your* software (Hint: We can’t. We won’t).

Sun, yes your hardware is sexy. How about a few blog entries from Jonathan showing how you are sorting out the software stack instead of endless “Look how sexy our new hardware is” posts.

I few years ago I fought Exchange to get the open standards of JES based on promises from Sun. Now, I don’t think I care tbh. Management feel that Sun have shafted us and I’m beginning to agree

They came, they swore a bit, they finally managed to upgrade us to v4.15F. Great. Problem solved. Or so we thought. A few days later we see

Mar 2 02:45:02 mailstore2 ufs: [ID 913664 kern.warning] WARNING: /data/mailstore2/msg/students4: unexpected allocated inode 2050084, run fsck(1M) -o f
Mar 2 02:45:02 mailstore2 ufs: [ID 913664 kern.warning] WARNING: /data/mailstore2/msg/students4: unexpected allocated inode 2050095, run fsck(1M) -o f

Bugger.
Then we get

Mar 6 10:08:51 mailstore2 ufs: [ID 913664 kern.warning] WARNING: /data/mailstore2/msg/staff2: unexpected allocated inode 6310183, run fsck(1M) -o f

And so it continues. We are still seeing some form of filesystem rot…and if anything, it’s now worse than it was Due to the amount of downtime we have already had on this so-called resilient HA system (Sun, this is getting harder and harder to sell to management btw, particularly as it seems impossible to even get a renewal quote for the bloody thing. Exchange is creeping up…) it was decided by some expertsmanagers that we should shut the system down and fsck a filesystem each week during our normal at risk period. A dangerous idea given we have already had one kernel panic but understandable given the bad image the system already has.

A couple of fscks pass, and we are still seeing the occasional error when we realise that one of the disks reported as needing fscking is the one we did last week. Shit. Disks are now corrupting at a faster rate than we are fixing them. Around this time, Sun send us info on bugid 102815. Bingo. That sounds like the thing. Except it seems to suggest the bug is in the firmware that we paid sun to move us to a month earlier :-/ Annoying is one word (and it doesn’t completely explain our earlier problems).

Rather rapidly, a whole days downtime is arranged for the following Sunday. We plan to come in, fsck all the filesystems on the arrays and then do the firmware upgrade which this time is a simple upgrade that can even be done live (ha!). Excellent.

Sunday comes, fscks take around 5 hours to complete cleanly (after swearing a bit at bugid 4775289 grrrr) and we start the firmware upgrade. First bad sign, it fails the controller over as it is meant to but the secondary then refuses to start up and do anything so all the disks go offline (and then report they need a fsck – argggghh). Still, the prompt reappears so we continue with the doc that says

Verify that the firmware upgrade succeeded by examining the firmware
revision again. Type the following command at the sccli> prompt:

show inquiry

and verify that the firmware revision is now reported as 415G.

Our still shows 415F. And to top it off, we now can’t retry the upgrade as one of the controllers is now flagged as “detected” instead of “enabled” and appears to have no serial number reporting in the gui…this is not a documented state A quick phonecall to sun (thank god for our elevated support at the moment following my email) and a rather unfazed support guy suggests we ignore the instructions and reset the controllers. This half worked….in that it did indeed make the controllers go offline. 15 minutes later they haven’t come back online (this should take “a couple of minutes”). Arse. We ring again. “try pulling the power from the array?”. Sigh. We do this and finally, it wakes up, realises it has new firmware and boots. Phew….

Service restored with 8 mins of our booked outage left. A full 8 hours of downtime *again*. It remains to be seen if we have finally nailed the problem or if we are still slowly dropping bits of filesystem at random

Google maps is indeed v. cool. Gmail likewise, and with the growing popularity of the other google apps with students at the university I work at we are repeatedly being asked why comms express isn’t more AJAX enabled.

Google have raised the bar on web apps way above Suns offerings. Any chance Sun are going to get anywhere near the levels of usability that google can offer our students for nothing? It’s getting harder and harder to justify our Sun solution – and this time it isn’t just Exchange knocking at the door…

Interesting you use a Blackberry – do you use that against the JES stack (and do you have cal access on it?). Another thing we are looking at that is easy (and cheapish) on Exchange and a minefield with Sun. Sun UK don’t even seem able to suggest anything to us that comes with a support contract that isn’t US working hours

In other news, michs car was fixed – under warranty – with no arguing! I’m amazed!

I feel a little let down now – I was looking forward to one last row with the bloody place