Tag Archives: 2

An Introduction to Data Blending – Part 2 (Hans Rosling, Gapminder and Data Blending)

Readers:

In Part 1 of this series on data blending, we began to explore the concepts of data blending as well as the life-cycle of visual analysis.

Today, in Part 2 of this series, we will dig deeper into how data blending works.

Again, much of Parts 1, 2 and 3 are based on a research paper written by Kristi Morton from The University of Washington (and others) [1].

You can learn more about Ms. Morton’s research as well as other resources used to create this blog post by referring to the References at the end of the blog post.

Best Regards,

Michael

Data Blending Overview

Data Blending allows an end-user to dynamically combine and visualize data from multiple heterogeneous sources without any upfront integration effort. [1] A user authors a visualization starting with a single data source – known as the primary – which establishes the context for subsequent blending operations in that visualization. Data blending begins when the user drags in fields from a different data source, known as a secondary data source. Blending happens automatically, and only requires user intervention to resolve conflicts. Thus the user can continue modifying the visualization, including bringing in additional secondary data sources, drilling down to finer-grained details, etc., without disrupting their analytical flow. The novelty of this approach is that the entire architecture supporting the task of integration is created at runtime and adapts to the evolving queries in typical analytical workflows.

A Simple Illustrative Example

In this section we will discuss a scenario in which three unique data sources (see left half of Figure 1 below for sample tables) are blended together to create the visualization shown in Figure 2 below. This is a simple, yet compelling mashup of three unique measures that tells an interesting story about the complexities of global infant mortality rates in the year 2000.

Figure 1

 

Image: Kristi Morton, Ross Bunker, Jock Mackinlay, Robert Morton, and Chris Stolte, Dynamic Workload Driven Data Integration in Tableau. [1]

In this example, the user wants to understand if there is a connection between infant mortality rates, GDP, and population. She has three distinct spreadsheets with the following characteristics: the first data source contains information about the infant mortality rates per 1000 live births for each country, the second contains information about each country’s total population, and the third source contains country-level GDP. For this analysis task, the user drags the fields, “Country or Area” and “Infant mortality rate per 1000 live births”, from her first data source onto the blank visual canvas. Since these fields were the first ones selected by the user, then the data source associated with these fields becomes the primary data source.

This action produces a visualization showing the relative infant mortality rates for each country. But the user wants to understand if there is a correlation between GDP and infant mortality, so she then drags the “GDP per capita in US dollars” field onto the current visual canvas from Data Table A. The step to join the GDP measure from this separate data source happens automatically: the blending system detects the common join key (ı.e. “Country or Area”) and combines the GDP data with the infant mortality data for each country. Finally, to complete her analysis task, she adds the “Population” measure from Data Table B, to the visual canvas, which produces the visualization in Figure 2 below associated with the blended data table in Figure 1.

 

Figure 2

Image: Kristi Morton, Ross Bunker, Jock Mackinlay, Robert Morton, and Chris Stolte, Dynamic Workload Driven Data Integration in Tableau. [1]

Hans Rosling, Gapminder and Data Blending

The Gapminder World interactive graph below shows how long people live and how the number of children a woman has is affected by how much money they earn using different data sources.

Gapminder World for Windows

Image: Hans Rosling’s Wealth and Health of Nations (Gapminder.org) [2]

Hans RoslingIn the screenshot above, the y-axis shows us Children per women (total fertility) . The x-axis shows us Income per person (GDP/capita, PPP$ inflation-adjusted). The series data points (the bubbles) show us population for each country. If you were to click the Play button, you would see as an interactive “slide show” how countries have developed since 1800.

This demonstrates the flexibility of the data blending feature, namely that users can dynamically change their blended views by pivoting on different data sources and measures to blend in their visualizations.

In the screenshot below, Mr. Rosling explains how to use the interactive Gapminder World application.

Also, Mr. Rosling has provided Gapminder World Offline, which you can use to show animated statistics from your own laptop! It can be run on Windows, Mac and Linux. Here is a link to the download installation page on the Gapminder.org site.

And here is a link to the PDF for the Gapminder World Guide show above.

Gapminder World Guide

Image: Hans Rosling’s Gapminder World Guide (PDF) [2]

Next: Benefits of Blending Data

——————————————————————————————————–

References:

[1] Kristi Morton, Ross Bunker, Jock Mackinlay, Robert Morton, and Chris Stolte, Dynamic Workload Driven Data Integration in Tableau, University of Washington and Tableau Software, Seattle, Washington, March 2012, http://homes.cs.washington.edu/~kmorton/modi221-mortonA.pdf.

[2] Hans Rosling, Wealth & Health of Nations, Gapminder.org, http://www.gapminder.org/world/.

Dr. Dobbs Journal: The Corruption of Agile (The Agile Holocracy) – Part 3 of 3

Readers:

In my last post, I included a recent interview Andrew Binstock, Editor in Chief of Dr. Dobbs Journal, had with Dave Thomas about the corruption of Agile and how many people are misunderstanding what the Agile Manifesto really said.

Allen HolubToday, I am continuing with the third and final part of my series titled The Corruption of Agile by highlighting thoughts from one of the key thinkers in the area of agile, Allen Holub. Allen is a consultant, trainer, and speaker, specializing in Lean/Agile process infusion, Agile/OO architecture, and cloud-based web-application development. He’s worn every hat from grunt programmer to CTO, and has written hundreds of magazine articles for various technical publications and a dozen books, including Holub on Patterns: Learning Design Patterns by Looking at Code. He is a regular speaker at various conferences, worldwide.

In a recent blog entry in Dr. Dobbs Journal titled The Agile Holocracy, Allen noted that agile is a culture, not a set of practices. It is upper management’s job to establish that culture, and then let it work. A holocratic organization is one of the better ways to do that.

Mr. Holub notes that the word holocracy popped up in a recent Dilbert cartoon, used incorrectly as a synonym for both a flat organizational structure and, as the pointy haired boss says, “wander[ing] around.” Nonetheless, full-on agile organizations have many of the characteristics of a holocracy, so Allen thought it was worth exploring the concept.

The term holocracy was coined by Artur Koestler in his 1967 book, The Ghost in the Machine, which is actually a discussion of the human brain. Koestler’s thinking applies to social organizations as well, however. A holocracy is made up of autonomous, self-reliant units called holons (from the Greek word for “whole”). The holons are simultaneously independent from and dependent on the organization that they comprise. That is, a holocracy is simultaneously a whole and its parts.

Kosetler uses our own brains as an example. The cerebellum (the “reasoning” brain) is largely independent on the amygdala (the so-called “lizard brain”). These two sub-brains operate independently under their own rules, but interact in a way that allows the whole system to behave both coherently and in a way that you couldn’t predict from looking at a single subsystem. This is a kind of emergence; the coherent behavior that arises from the complex interactions of the rules and strategies of the independent holons is more sophisticated than behavior of the parts. The ghost is the thing that you can’t predict just by looking at the parts.

Koestler also points out that when one subsystem dominates, the whole system can become pathological. When your lizard brain hears a loud noise and says “panic!,” your cerebellum counters with, “no, that’s just an electric guitar and a gazillion-watt amp” and overrides it. The synthesis is a feeling of excitement that the concert’s starting. If the lizard brain wins, everybody in the stadium gets trampled.

The notion of holocracy works well in agile organizations. In fact, I’d say that some level of holocratic structure is essential. Independent units — teams or individuals — are responsible for specific organizational roles, and interact with one another in a synergistic (to use an overused word) way. A person is holon, and so is a team. Each holon is both independent and part of the whole. Each holon is responsible for itself, but is also responsible for the success of the whole organization. The holons carry out well-defined roles, and are responsible for doing whatever they have to do to succeed in those roles. Nobody orders anybody around. The teams and the people are responsible for defining and executing their own tasks.

Getting back to Dilbert, holocratic organizations do have a flat hierarchy. They are peer-to-peer networks, not trees, and there’s no place for any sort of “middle management.” The executives are the organization’s lizard brain: They set the direction and support the broad culture. The teams are the cerebellum: Thinking out the details and getting the processes to work. The synthesis of those roles yields a functional agile organization. If either holon is dominant (that is, if agile practice exists without a supporting culture, or if the executives perpetuate command-and-control, or if the teams get to do everything they want without regard to budget or scope), everybody gets trampled.

There isn’t even a “CEO” in the captain-of-the-ship sense. Instead, the holons (both people and teams) communicate with each other in order to tune what they’re doing. In the same way that people talk to each other, teams interact horizontally. For example, teams can “inject” members into other teams. Spotify, for example, has a “guild” system. All the database folks from all of the teams get together occasionally for Database-Guild meetings, and so forth. Think of this arrangement as a database-guild holon that injects members into the development-team holons.

Holocratic organizations aren’t fantasies. Probably the most well-known is W.L. Gore, Inc., as described in Malcolm Gladwell’s Tipping Point. There are no formal titles and no hierarchical structure. People have “sponsors,” but nobody reports to anybody.

The main objections to the viability of a holocracy seem to come from people who confuse holocracy with anarchy — everybody doing whatever they want. That’s not what I’m describing. A holocracy is highly disciplined. You take your responsibilities seriously, and work hard both to be effective and to make everybody else successful. Communication and collaboration are central to both functions, so everybody does that.

Similarly, holocracy is not Communism. The people who contribute more are rewarded accordingly. However, compensation has to be transparent and rational. Secrets undermine the collaborative nature of the system.

Dysfunction happens when one holon becomes dominant. An out-of-control “executive” holon, for example, can destroy the agility of the organization. I’ve seen teams, departments, and whole companies brought to their knees by pathological “leaders” intent on dominating, who think that somebody has to “run” things. The mandates that come from these people do nothing but add dysfunction. I’m not talking about micromanagement, which is also a sort of disease. Dysfunction happens any time somebody can’t do something because they need permission, or won’t do something because they’re afraid of some repercussion. The holons (teams and individuals) must be empowered to fulfill their roles.

Of course, there’s no reason for some of the holons not to be tasked with the health of others. For example, one responsibility of the “executive” holon is to assure that company has an effective internal culture. If a team becomes dysfunctional, the “executive” holon’s job is to help the team heal, not by ordering people around, but by providing training, mentorship, etc. If that doesn’t work, though, the “executive” holon could act like an antibody. The immune system is part of a healthy brain. The core value in a holocracy is trust. Hierarchical command/control organizations are built on mistrust. They assume that nobody beneath you in the hierachy is truly self sufficient and competent, so they impose paternalistic governance practices that force these assumed-incompetent players to behave. A holocracy trusts everybody to do their work in the best way possible.

Since trust and assumed competence are also firmly ingrained in the Agile Principles, there’s a good match here. Everybody’s got to be a grownup, though. I once heard Aliestar Cockburn answer the criticism “your methods won’t work in the real world because they require nothing but A players” by saying that making good software always requires A players, regardless of your process. I agree.

As a final note, Holacracy.org adapted Kosteler’s notions into a social/organizational milieu, and serves as an interesting example of another functional variant of an holocracy. Everybody in the Holacracy.org is literally (in the legal sense) a partner. There are no bosses, and there is no CEO. I’m not a huge fan of Holocracy.org’s formalization, but they’re prominent enough to deserve mention. For one thing, I’m offended by the way that Holacracy.org has trademarked Kosetler’s term. The trademark isn’t enforceable, given prior use, but they have no right to set themselves up as the arbiter of somebody else’s idea. It’s like somebody trademarking “agile.” Moreover, I don’t particularly like their fixed rules (documented in a Constitution), which seems to me to fly in the face of the core principle of self organization.

I’ve said this before, but agile is a culture, not a set of practices. It is upper management’s job to establish that culture, and then let it work. A holocratic organization is one of the better ways to do that.

———————————————————————————–

Source: Allen Holub, The Agile Holocracy, Dr. Dobbs Journal, March 14, 2014, http://www.drdobbs.com/architecture-and-design/the-agile-holocracy/240166629.

 

Dr. Dobbs Journal: The Corruption of Agile (or What the Agile Manifesto Really Said) – Part 2 of 3

Readers:

In my last post, I included a recent article from Andrew Binstock, who is the Editor in Chief of Dr. Dobbs Journal. Mr. Binstock expressed his concerns about how we are corrupting Agile.

Dave ThomasToday, I am continuing with Part 2 of my The Corruption of Agile series by highlighting thoughts from one of the founding authors of the Agile Manifesto, Dave Thomas. Dave (photo, right) is a coauthor of the legendary Pragmatic Programmer book, one of the authors of the Agile Manifesto, a publisher, speaker, and the man who brought Ruby to the masses.

In a recent interview with Mr. Binstock, Dave discussed what he feels is wrong with Agile. But, before we get started, let’s review what the Agile Manifesto really said. This is verbatim from the Agile Manifesto website.

 

Manifesto for Agile Software Development

We are uncovering better ways of developing
software by doing it and helping others do it.
Through this work we have come to value:

Individuals and interactions over processes and tools
Working software over comprehensive documentation
Customer collaboration over contract negotiation
Responding to change over following a plan

That is, while there is value in the items on
the right, we value the items on the left more. [2]

The Corruption of Agile – Andrew Binstock’s Interview with Dave Thomas [1]

AB: Let me move to the Agile Manifesto. You were one of the original signatories…
DT: Yes, yes. There were 17 of us that met in Snowbird in Utah in February 2001; and we came up with the four values and the common ground and practices. The way we came up with it was not something I’d ever experienced before. There were 17 people and everyone had their own ax to grind and their own particular viewpoint on what was best. And within a couple of hours, we’d come up with a way of working that let us discuss the principles behind what we were doing without going into the details.So the framework itself, the actual four values, came out in half a day. It just kind of came. I think everyone kind of knew what they wanted to say. I think it was actually Martin Fowler and I — everyone else was having lunch — we were kind of sitting at a whiteboard just chatting through things and we came up with the “we value A over B” formulation, which as it turns out, has worked out really well.
AB: I think that perhaps a limitation is that a lot of people have interpreted it as “no B whatsoever.”DT: Indeed! And, in fact, we recognized that might happen at the time, and we put in the little phrase at the bottom: “whereas we value B, we value A more.” So, we thought that would happen and we tried to stop it, but, of course, we…AB: …can never control…

DT: Yes, that was the depressing thing for me. I came away from Snowbird feeling really happy. And within a month, I realize it wasn’t going to work. And I have not been involved in anything even called “Agile” until last month.
AB: Last month?
DT: That’s right, last month. I have not gone to any Agile conferences. I have not joined the Agile Alliance.
AB: Why is that?
DT: Because it got immediately productized in many different ways. The whole point, to my mind, of the Agile Manifesto is that it’s a set of personal practices that may scale to team level. You do not need a consultant to show you how to do that. It may help to have someone facilitate, but you do not need a consultant. And yet immediately what happened was that everyone and their dog hung out an Agile shingle and the whole thing turned into a branding exercise.
AB: Yes.
DT: And so now in the software world, if you’re selling something and it doesn’t have the word “Agile” somewhere in the title, it’s not good. And I hate that because we had something that was…it was my best attempt at the time to present a set of free-standing, self-contained values that could be applied by all. And now it’s just become mush. It’s become devalued to the point where it doesn’t mean anything.
DT: I also have a pure English-language abhorrence to the people who use Agile as a noun. As in “I do Agile.” No you don’t. It’s like saying “I do orange.”
AB: When you were meeting at Snowbird, was there a feeling there that you were doing something important that was going to change software?DT: Good question.AB: The fact that you chose the term “manifesto” suggests that you were planting a flag.DT: There was a lot of discussion at the time about what it should be called. And, to be honest, I can’t remember to what extent we were thinking of posterity when we came up with the word “manifesto.” Even the word Agile was debated a lot.

AB: That’s what I’ve heard.

DT: I think we discussed “lightweight.” There was some sense that this was going to be bigger than the 17 of us because we agreed on the last day that Ward [Cunningham] was to go create a site that would allow people to sign up. Ward is someone who is always building community. And I think we all went along thinking that would be a really good way to spread the word. But the reality was that I don’t think any of us were particularly marketing people. The fact that it took off as quickly as it did was that we just happened to say the right thing at the right time. We didn’t actively push it particularly, it just kind of pushed itself or actually pulled itself.

Next: Allen Holub and The Agile Holocracy

Bonus Section: The 12 Principles behind the Agile Manifesto

For you folks that have never read the actual Agile Manifesto, I have provided a copy of its guiding principles here for your review.

We follow these principles:

Our highest priority is to satisfy the customer
through early and continuous delivery
of valuable software.

Welcome changing requirements, even late in
development. Agile processes harness change for
the customer’s competitive advantage.

Deliver working software frequently, from a
couple of weeks to a couple of months, with a
preference to the shorter timescale.

Business people and developers must work
together daily throughout the project.

Build projects around motivated individuals.
Give them the environment and support they need,
and trust them to get the job done.

The most efficient and effective method of
conveying information to and within a development
team is face-to-face conversation.

Working software is the primary measure of progress.

Agile processes promote sustainable development.
The sponsors, developers, and users should be able
to maintain a constant pace indefinitely.

Continuous attention to technical excellence
and good design enhances agility.

Simplicity–the art of maximizing the amount
of work not done–is essential.

The best architectures, requirements, and designs
emerge from self-organizing teams.

At regular intervals, the team reflects on how
to become more effective, then tunes and adjusts
its behavior accordingly. [2]

———————————————————————————–

Sources:

[1] Andrew Binstock, Dave Thomas Interview: The Corruption of Agile; Ruby and Elixir; Katas and More, Dr. Dobbs Journal, March 18, 2014, http://www.drdobbs.com/architecture-and-design/dave-thomas-interview-the-corruption-of/240166688?pgno=3.

[2] Kent Beck et al, Manifesto for Agile Software Development, The Lodge at Snowbird Ski Resort, Utah, February 11-13, 2001, http://agilemanifesto.org/.

 

 

Bryan Redux #2: Removing Sections of a Report via URL API

Readers:

Here is another blog re-post from my friend, Bryan Brandow’s MicroStrategy site.

If you want to directly visit Bryan old site, the URL is http://www.bryanbrandow.com.

Best Regards,

Michael

Bryan originally posted this on July 22, 2011.

Removing Sections of a Report via URL API

WhBryan Brandowile working with the Web SDK to try to make a small customization, I stumbled on a pretty useful set of URL API codes that allow you to quickly modify the Report Page by removing various elements.  While such customizations are common, I wasn’t aware that they were available out of the box.  I would think that these would be very useful in doing simple linking to reports via an IFrame from another application, such as a Portal.

The trick to get the URL for a Report by right clicking it in Web, choosing Properties, and then copying the Link at the bottom.  Now, locate the src section of the URL:

ASP: &src=Main.aspx.4001 

JSP: &src=mstrweb.4001

and add in one of the transforms just before the .4001.

Available transforms that I’ve found:

reportNoHeader

reportNoHeaderNoFooter

reportNoHeaderNoFooterNoPath

reportNoHeaderNoFooterNoPathNoToolbar

Example:

ASP: &src=Main.aspx.reportNoHeader.4001

JSP: &src=mstrweb.reportNoHeader.4001

Unfortunately, these are actual transforms and not flags, so you can’t mix and match to fit your needs (for example, if you only want to hide the toolbar, reportNoToolbar will NOT work).  Those are the only 4 that you can use that I’ve found, but they may be handy in a pinch and best of all, not require customization work to use.

Update:

Another method of doing this that does let you pick and choose and supports documents as well:

&hiddenSections=header,footer,path,dockTop

Place that code in the URL, for example (Note: I have shown this on multiple lines. You should paste this in as one long line with no line breaks):

http://webserver/Microstrategy/asp/Main.aspx?Server=ISERVER&Project=PROJECT&Port=0&evt=2048001
&hiddenSections=header,footer,path,
dockTop&src=Main.aspx.2048001
&visMode=0&documentID=04AACFE445AD69198676C8AD56245118
&currentViewMedia=2

List of options (case-sensitive):

  • header
  • path
  • dockTop
  • dockLeft
  • dockRight
  • error
  • content
  • dockBottom
  • footer

Bryan’s Blog Entry Link:  http://www.bryanbrandow.com/2011/07/removing-sections-of-report-via-url-api.html

Commentary: Some Thoughts on my MicroStrategy v9.4.1 Upgrade Installation on my Laptop – PART 3

Readers:

I am back to continue and finish this three-part commentary about installing MicroStrategy v9.4.1 on my laptop.

Just also wanted to let you know that v9.4.1 Hotfix 2 was released on 02/12/2014 and is now available. I recommend you get your v9.4.1 GA version all up and running properly before you consider installing the Hotfix.

Best regards,

Michael

Reviewing Upgrade Prerequisites

Before you begin upgrading your MicroStrategy system, it is always a good practice to review the MicroStrategy Readme document so that you are aware of any changes from previous releases. You should also review the system prerequisites outlined in the Planning Your Installation chapter of the MicroStrategy Installation and Configuration Guide.

If you do not review the MicroStrategy hardware and software requirements before upgrading, you may experience problems with the upgrade.

Database and Driver Requirements

Refer to Certified and Supported Configurations in the MicroStrategy Readme for updated information about specific database and driver combinations certified by MicroStrategy.

System Sizing Guidelines

There are several factors to consider when you initially set up your MicroStrategy system. These factors include the number of users that will access the system, report complexity, and whether or not you should employ caches. You should periodically re-evaluate your system and update it based on actual system performance and use.

In particular, before updating your metadata (see the Update the Metadata section below), MicroStrategy recommends that you have an amount of free disk space equal to at least twice the on-disk size of the metadata database.

A complete discussion of system sizing guidelines is beyond the scope of this blog entry. Refer to the Planning Your Installation chapter of the MicroStrategy Installation and Configuration Guide for the latest details about sizing your system.

Due to performance improvements and enhancements, MicroStrategy version 9 may require more memory than version 8 for comparable functionality. In particular, if your MicroStrategy 8.x system is running on Windows and is approaching the 3 GB Windows memory limit, you may need to upgrade your Intelligence Server machines. For more information on MicroStrategy memory recommendations, see the system requirements in the MicroStrategy Readme and the Tuning chapter in the System Administration Guide.

Privileges and Access

Before upgrading, ensure you have the following:

  • If you are installing on a Windows system, you must have a login account with administrative privileges for the domain or target machine.
  • MicroStrategy Intelligence Server installation files. You can access the installation files from a disk or from a network location.
  • Write permissions in the installation directory; otherwise the installation/upgrade process fails.
  • If you have purchased a CPU-based MicroStrategy license and are installing on UNIX or Linux, you need root access permissions for installation.
  • A license key from MicroStrategy for the version of the MicroStrategy software that you are installing.

Checking for Supported Data Types

MicroStrategy Analytics Enterprise supports a wide variety of data types for each supported warehouse database. However, some pre-Analytics Enterprise projects may contain data types not supported in MicroStrategy Analytics Enterprise. If a project containing columns with unsupported data types is upgraded, the data types for those columns are assigned as “reserved,” and proper data types are not assigned in temporary tables. This affects report execution.

Before proceeding with the upgrade, you must ensure that all data types assigned in the pre-Analytics Enterprise project are supported in MicroStrategy Analytics Enterprise.

See the MicroStrategy Project Design Guide for a listing of the supported data types for each database type and additional information about changing to supported data types.

Backing up the Metadata

Although the MicroStrategy installation process itself does not affect your project’s metadata, MicroStrategy recommends that you back up your metadata before any significant installation or upgrade. In most major MicroStrategy upgrades, a metadata update is required for all the pre-existing projects in your metadata. Once you update your metadata project, you cannot revert that metadata to a previous version. Therefore, MicroStrategy strongly recommends that you perform a full database backup of your original metadata prior to the upgrade.

MicroStrategy strongly recommends that you also tape backup, image, or ghost the production server before upgrading.

If you want to keep an old MicroStrategy Tutorial metadata repository and warehouse from a previous MicroStrategy version, rename the Microsoft Access files or move them to another location; otherwise, they are overwritten during the installation process. The Access files are installed by default in the MicroStrategy\Tutorial Reporting folder.

Updating the Project Metadata

MicroStrategy requires that you use the Configuration Wizard to update a metadata project created in a pre-Analytics Enterprise version of MicroStrategy to the latest version.

Be aware of the following:

  • If you are upgrading a MicroStrategy 8.x metadata that is stored in a DB2 UDB for z/OS database, refer to MicroStrategy Tech Note TN32695.
  • For assistance with updating MicroStrategy metadata projects from versions prior to MicroStrategy version 8.1.0, contact MicroStrategy Technical Support.

MicroStrategy requires that you update projects through an Intelligence Server connection (3-tier). Upgrading your project using a direct ODBC connection (2-tier) is not supported.

If you do not upgrade the metadata to the latest version, certain features will not work as expected. For example, if MicroStrategy Web Analytics Enterprise connects to a pre-Analytics Enterprise metadata through an Analytics Enterprise server, Change Journaling, Distribution Services, and some Report Services enhancements may not be available.

Downgrading Metadata Projects

Downgrading a MicroStrategy metadata or project to any previous product version is not supported. Once you update the project metadata to the latest version, you cannot downgrade to earlier product versions. Therefore, backing up the metadata is an essential step in the upgrade process as it allows you to revert to a backup version of the metadata, if necessary, to obtain pre-update versions of the projects it contains.

Configuring an Upgrade Test Environment

Your MicroStrategy environment includes multiple variables, such as security requirements, performance requirements, and VLDB settings, that are unique. MicroStrategy cannot anticipate all the ways these variables may interact with the upgrade process. Thus, MicroStrategy recommends you create a test environment and upgrade that environment first, then thoroughly test the upgraded installation. Once the tests are complete, then upgrade your production environment. This ensures that the upgrade of your production environment proceeds smoothly and any unexpected difficulties do not require additional downtime.

I will post a blog in the near future about testing your upgraded environment.

If you do not want to create a test environment, MicroStrategy recommends that you create and save an Integrity Manager integrity test baseline of your reports and documents. You can then execute an integrity test against this baseline when the upgrade is complete, to ensure that the upgrade has not altered any of your report results. For detailed information about using Integrity Manager to execute integrity tests, see the Integrity Manager chapter of the MicroStrategy System Administration Guide.

Best Practices for Configuring an Upgrade Test Environment

MicroStrategy recommends that you follow these best practices for configuring your upgrade test environment:

  • Do not modify any existing configuration objects. If you need additional configuration objects for testing, you can either create additional objects, or duplicate an existing object and modify it. This applies to database instances, connections and logins, security filters, users and user groups, and security roles.
  • If your production environment is clustered, then your test environment should also be clustered.
  • If your test and production data warehouses have different database table prefixes, make sure you are using the correct prefixes in the test environment’s Warehouse Catalog.
  • Create an integrity test comparing reports from the upgraded test environment with the same reports in the production environment, so that you can easily see where any differences are.
  • If possible, plan to execute data integrity and performance load tests against the production warehouse. This ensures that the test scenarios are as representative of the production environment as possible.
  • If you are creating reports and documents specifically for an upgrade integrity test, create those reports and documents before you duplicate the production metadata.
  • If you are using connection mapping for users to access the data warehouse, check to be sure that all users can log in to the test data warehouse, since user passwords may differ between the test warehouse and the production warehouse.

One way to manage this is to create a new generic database login, and then use the following sample Command Manager script to change users’ connection mappings to use this new login:

ALTER CONNECTION MAP FOR USER “

username” DBINSTANCE “production_warehouse_instance” DBLOGIN “test_login” ON PROJECT “project“;

For steps to use Command Manager, see the Command Manager Help, or the Command Manager chapter of the MicroStrategy System Administration Guide.

  • If you are using Narrowcast Server, make sure that the database copy of the Narrowcast repositories is not used when setting up the Narrowcast Server test environment. Instead, make a copy of the repositories with the Copy Repository utility included with Narrowcast Administrator and use this copy. This ensures that the test environment does not accidentally refer to a production server. For detailed instructions on creating a copy of the Narrowcast repositories, see the Narrowcast Server Upgrade Guide.

High-level Steps to Configure an Upgrade Test Environment

To ensure that your tests accurately reflect the upgrade experience, the upgrade test environment should reflect the production environment as closely as possible.

To Configure a Test Environment

  1. Set up the hardware for the environment. MicroStrategy recommends that this hardware duplicate the configuration of the production environment as closely as possible.
  2. Install your current version of MicroStrategy in the test environment.
  3. Using the Project Duplication Wizard, duplicate the production metadata into the test environment. For instructions on using the Project Duplication Wizard, see the Managing Your Projects chapter of the MicroStrategy System Administration Guide, or see the Project Duplication Wizard Help.
  4. Make sure that your test environment Intelligence Server is connected to your test environment metadata, and not your production metadata.
  5. If you do not intend to execute your tests against a production warehouse, duplicate the production warehouse, and ensure that the test environment points to the duplicate warehouse and not the production warehouse.
  6. Upgrade the test environment.
  7. Test the upgrade. Again, a future blog topic.

Upgrade Deployment Tests

Deploying the upgrade involves installing, activating, configuring, and running the upgrade processes for Intelligence Server, MicroStrategy Web Server, and MicroStrategy Mobile Server, as well as for the metadata, Narrowcast Server, and Enterprise Manager data repositories. These changes, as well as any other procedures that alter the production environment, should be tested when setting up the test environment.

Deployment tests should be performed by MicroStrategy administrators who normally have the responsibility of tuning and monitoring the MicroStrategy installation.

 

Reference Materials

Some detailed information about installing and configuring MicroStrategy products is beyond the scope of this blog entry and can be found in the MicroStrategy Installation and Configuration Guide. The MicroStrategy Installation and Configuration Guide provides detailed procedures on installing and configuring your MicroStrategy system. It also includes important information about installing, deploying, and configuring MicroStrategy Universal products.

In addition, the MicroStrategy Readme contains information about the new products, new features, and bug fixes available in MicroStrategy Analytics Enterprise.

For detailed instructions for upgrading Narrowcast Server, refer to the Narrowcast Server Upgrade Guide.

Commentary: Some Thoughts on my MicroStrategy v9.4.1 Upgrade Installation on my Laptop – PART 2

MicroStrategy Platform v9.4.1

Upgrade best practices

Review the following recommendations to help ensure the success and stability of your MicroStrategy system and projects when upgrading to the latest version of MicroStrategy.

  1. Follow the upgrade order and recommendations outlined in this section, in particular the upgrade checklist found at The Upgrade Process Checklist in the section below. In particular, always upgrade Intelligence Server prior to upgrading client applications such as MicroStrategy Web or Developer.
  2. Create an upgrade test environment by duplicating your production environment and production metadata. Upgrade this test environment and test it before upgrading your production environment.
  3. Do not downgrade MicroStrategy products or components on a machine to previous versions if you have already installed the most recent version of another MicroStrategy product on that machine.
  4. All MicroStrategy products on a machine must use the same version of MicroStrategy. Do not install or upgrade only some MicroStrategy 9.3.1 products on a machine containing older versions of other MicroStrategy products.
  5. Avoid installing MicroStrategy products using services such as Windows Terminal Services, which create a virtual session on the host machine. Always install MicroStrategy directly on the server machine’s physical interface, or by using a remote connection tool (such as Microsoft Netmeeting or Virtual Private Network) that takes full control of the server machine’s interface.
  6. If you are using clustered Intelligence Servers, then to retain stability in your Intelligence Server cluster while upgrading, shut down Intelligence Server on all nodes in the cluster before proceeding with the upgrade. For more information about clustering Intelligence Servers, see the Clustering chapter in the System Administration Guide.
  7. Every node in the MicroStrategy cluster must run the same version of MicroStrategy for the cluster to work properly.

The Upgrade Process Checklist

The upgrade process described in this section involves the following high-level steps. To help ensure a successful upgrade, follow these steps in the order they are presented in this section.

1. Prepare the MicroStrategy system and projects for upgrade

Preparing a MicroStrategy system for an upgrade involves reviewing information specific to your version upgrade, pre-upgrade information and prerequisites, checking for supported warehouse data types, and backing up the production metadata. It may also involve creating an upgrade test environment that duplicates your production environment.

2. Install and configure Intelligence Server Analytics Enterprise and Developer Analytics Enterprise on a test server

In this step, you install and configure MicroStrategy Intelligence Server Analytics Enterprise and MicroStrategy Developer Analytics Enterprise on a test server and then establish a connection to your production metadata.

3. Update the production metadata

In this step, you update the metadata version of your production projects using the test server environment.

4. Perform basic stability testing

In this step, you perform basic testing to ensure the stability and efficiency of Intelligence Server and your updated projects.

5. Install and configure Intelligence Server in the production environment

Once you are satisfied with the status of the latest version of Intelligence Server, and have updated the projects in your test environment, you install Intelligence Server in the production environment.

6. Install remaining MicroStrategy products in the production environment

With the latest version of Intelligence Server installed in your production environment, you now install and configure the remaining MicroStrategy products in your production environment.

7. Test the upgrade, and perform other post-upgrade tasks

After upgrading to the latest version of MicroStrategy, you perform several post-upgrade tasks such as testing the system, activating your installation, checking system licensing and functionality, managing user privileges, and optimizing your MicroStrategy system.

Next: Reviewing upgrade prerequisites

Tips & Tricks #2: In MicroStrategy Web, Report Execution Fails with Error ‘Results for this message cannot be retrieved from MicroStrategy Intelligence Server. This might be because the execution has failed. Please contact your administrator.’

I had this error the other day. Fortunately, I had just been reading about working set governor settings the other night to prepare for the CPA and MCE exams.

First, let’s discuss the issue.

Issue/Problem

In MicroStrategy Web, when I executed a report, I received the following error message (also, see screenshot below).

(Results for this message cannot be retrieved from MicroStrategy Intelligence Server. This might be because the execution has failed. Please contact your administrator.)

Results cannot be retrieved error message

Basically, what happened was my report request could not be processed. Looking through the MicroStrategy KnowledgeBase, it basically tells you to try to run the report again. If it still throws this error, contact your Administrator.

If you look at the Web Log on the Web Server, you will see an error similar to the following:

<record reset=`true`>
 <package>com.microstrategy.webapi</package>
 <level>SEVERE</level>
 <miliseconds>1157577081154</miliseconds>
 <timestamp>09/06/2006 14:11:21:154</timestamp>
 <thread>154</thread>
 <class>CDSSXMLReportServer</class>
 <method>GetExecutionResultsEx</method>
 <message>(Results for this message cannot be retrieved from 
MicroStrategy Intelligence Server. This might be because the
execution has failed. Please contact your administrator.) 
(com.microstrategy.webapi.MSTRWebAPIException)</message>
 <exception>com.microstrategy.webapi.MSTRWebAPIException: 
(Results for this message cannot be retrieved from 
MicroStrategy Intelligence Server. This might be because
the execution has failed. Please contact your 
administrator.)
 at com.microstrategy.webapi.
CDSSXMLReportServer.GetExecutionResultsCommon(Unknown Source)
 at com.microstrategy.webapi.
CDSSXMLReportServer.GetExecutionResultsEx(Unknown Source)
 at com.microstrategy.web.objects.
WebReportInstanceImpl.
getExecutionResults(Unknown Source)
 at com.microstrategy.web.objects.
WebReportInstanceImpl.pollForResults(Unknown Source)
 at com.microstrategy.web.objects.
WebReportInstanceImpl.populateUserReportCache(Unknown Source)
 at com.microstrategy.web.objects.
WebReportInstanceImpl.pollStatus(Unknown Source)
 at com.microstrategy.web.beans.
ReportInstanceProxy.pollStatus(Unknown Source)
 at com.microstrategy.web.beans.
ReportBeanImpl.localCollectData(Unknown Source)
 at com.microstrategy.web.beans.
ReportBeanImpl.doCollectData(Unknown Source)
 at com.microstrategy.web.beans.
AbstractWebBean.collectData(Unknown Source)
 at com.microstrategy.web.app.beans.
AbstractAppComponent.collectData(Unknown Source)
 at com.microstrategy.web.app.beans.
ReportFrameBeanImpl.collectData(Unknown Source)
 at com.microstrategy.web.app.beans.
AbstractAppComponent.collectData(Unknown Source)
 at com.microstrategy.web.app.beans.
PageComponentImpl.collectData(Unknown Source)
 at com.microstrategy.web.app.
MSTRWebController.processRequest(Unknown Source)
 </exception>
 <userName>Administrator</userName>
 <clientID>172.19.19.2</clientID>
 </record>

The DSSErrors.log file on the MicroStrategy Intelligence Server contains the following errors:

[Kernel][Error] MsiWorkingSet::PersistMsg(): 
failed to attach RI to msg, error 0x80003F79
 [Kernel][Error] CDSSServerMessage::put_OriginalRI: 
WSResultPool->AddRI for msg xxxx return error 0x80003F79
 [Kernel][Error] CDSSServerMessage::put_OriginalRI: 
WSResultPool->AddRI for msg xxxx return error 0x80003F79
 [Kernel][Error] CDSSServerMessage::GetReportInstance():
get_OriginalRI() return error 0x1

Now what?

Solution

The size of the report that was to be executed has 40MB report cache size while the ‘Maximum RAM for Working Set cache size’ is set as 102,400KB, as shown in the image below:

Governing Rules - Default - Working Set

The MicroStrategy Intelligence Server was unable to swap out the report instances of 40MB in the Working Set.  To resolve this issue, I needed to increase the size of Maximum RAM for Working Set cache to a higher value, for example 512,000KB.

What is the ‘Working Set’ Governor Setting?

When a user runs a report from MicroStrategy Web or Web Universal, the results from the report are added to what is called the working set for that user’s session and stored in memory on the Intelligence Server. The working set is a collection of messages that reference in-memory report instances. A message is added to the working set when a user executes a report or retrieves a message from his or her Inbox.

The purpose of the working set is to:

  1. Allow the efficient use of the web browser’s Back button.
  2. Improve web performance for manipulations.
  3. Allow users to manually add messages to the History List.

This setting is accessible via the MicroStrategy Intelligence Server Configuration as shown below:

Governing Rules - Default - Working Set

Working Set Governors

The ‘Working Set file directory’ is the location in the filesystem where the Report Instances may be persisted on disk. A report instance will be persisted on disk in binary format if its size exceeds the limit set by the ‘Maximum RAM for Working Set cache’ governor or none of the report instances in memory can be swapped to make room for the new report instance. The persisted report instance will be persisted as the <filename(GUID)>.po and may be reused if the report is invoked again.

The ‘Maximum RAM for Working Set cache’ is the governor that modulates the size of the WorkingSet Result Pool. The Maximum value is: 2147483647 MB, the Minimum value depends on version and is 200 MB in 9.3.1, and the Default value is: 200 MB. Note that if the value specified is greater then the machines memory it uses the default of 200 MB. The default value is usually sufficient, but if memory issues arise, as noted above, the setting can be increased. Increasing this setting means that the MicroStrategy Intelligence Server may be operating with a higher average memory footprint during its lifecycle, so proper tuning may be needed if memory usage becomes an issue.

Important Notes

  • There is no Working Set (WS) for a session created by the MicroStrategy Desktop client.
  • This is a MicroStrategy Intelligence Server configuration level setting, so it applies to all the projects and all the users and is not specific to a project. If these settings are changed, MicroStrategy Intelligence Server may need to be restarted.
  • The MicroStrategy Working Set is not the same as the Microsoft Windows Operating System Working Set.