Process Mining Bible – Questions and Answers

I presented a series of webinars around Process Mining together with Anne Rozinat from Fluxicon about Process Mining.

The first webinar was held at October 3rd and some very interesting questions from the attendees were raised. This blog post aims to provide answers to those that could not attend or that continuously had doubts about the possibilities of Process Mining.

Question: Imagine that a company has a BPM system and have many process running on that platform. Also this company has a BAM System to analyze the process. how can we introduce Process Mining integrated to this technologies ?

Answer: I’m going to expand the question from BPM system to any other system that model, design and implements business processes even those that are brought to life on real time (instance mode). There is an assumption that the process model that was implemented reflects 100% the reality as it occurred. That is absolutely wrong. There is no straight line from start to finish. It’s not a matter of exceptions occur, it’s a matter that work does not happen the way it was designed for.

Even if the process is highly structured one instance can stop and halt at some point, other can stop in a different activity. There are no equal instances. This means if you look to process execution as a heat map, you will find patterns, practices that differ from the “main success” scenario.

If the process provides degrees of freedom, or is designed and executed on instance mode you can compare if the social network is aligned with the process type. In this situation knowledge type is exploratory or exploitative, but if the social network configuration among the people that participate is strong central with a network star, knowledge flow is inappropriate and a solution will take much more time to achieve, because the flows are being centralized by a manager rather than the people that need information to act.

BAM (Business Activity Monitoring) tell you everything about how many requests are being handled, how much time does it take, workforce balancing, if the company is missing the the service level agreement, but does not shows you WHERE you are lagging, WHAT is the cause of missing the targets. process Mining is process oriented information. Thus, it shows the pain points and bring you awareness HOW to act in real time.

Question: You presented some methods to model the control-flow perspective (using the order of activities), but what can you say about the data-flow perspective? There are also techniques to model the data, to see how data changes during a process?

Answer: What Process Mining technology does it’s to reconstruct for each instance the execution path (the process model) and sum it up to deliver it with all the expectations, loopbacks, and unexpected paths that the process took in reality. Each instance has predefined data attached that is defined at the start of the project. Data can be transformed in analysis dimensions (roles, costs, and data). Thus if the focus is to understand how data flow occurs, it can be analyzed under that perspective.

Question: (I love this one because is out of typical process automation domain). Suppose there are 1000 steps in the assembly, and 3 are in the wrong order. Would I be able to find this using your tools without manually checking every detail?

Answer: Imagine you are analyzing an assembly line that fills up bottles of wine. The assembly line has built in sensors that records the operations, the errors, and calculates the OEE (Overall equipment effectiveness). Once Process Mining is agnostic, you can analyze the recorded data and figure it out in the millions of wine bottles, those that went into label placement twice. Thus it’s possible to discover duplication, because there is an error setting up the labeling machine and you need to reprocess the label task.

Question: Can we use process mining to simulate or analyze interactions among business processes. i.e. the points or activities in the processes that can cause negative effects in the business processes (multiple organizations or same organization)?

Answer: Sure. Today (finally) processes are not being recorded only only inside corporate boundaries. They exist and return back to the company that sparked it (initiating an order that goes to a supplier warehouse, then a forwarding agent, gets back to the company, the supplier sends an invoice and hopefully there is a payment). Once there is a common key across multiple systems, it’s possible to mine the processes and get answers independently the company that performed some activities.

Question: The processes you demonstrated today are lasagna type processes. There are on the other hand also processes  that have a lot of variation in process flow and little numbers of instances. e.g. in Healthcare. What kind of help can PM give in these types of processes?

Answer: Contrary of some opinions that darkness can see the ray of light if the process is structured, there are lots of REAL world applications under Adaptive Case Managment. Here are some pointers:

• Application of Process Mining in Healthcare – A Case Study in a Dutch Hospital – R.S. Mans, M.H. Schonenberg, M. Song, W.M.P. van der Aalst, and P.J.M. Bakker;
• Beyond X-Raying a Care-Flow: Adopting Dierent Focuses on Care-Flow Mining Filip Caron, Jan Vanthienen, Jochen De Weerdt, and Bart Baesens;
• Finding Structure in Unstructured Processes: The Case for Process Mining – W.M.P. van der Aalst and C.W. Gunther
• Monitoring Unmanaged Business Processes – Nirmal K. Mukhi

Question: In what phase of a project would you recommend process mining? Is it important to have a certain level of CMM with steady and well knows processes?

Answer: Maturity models can be a useful to help people to make decisions, BUT, we can have a scenario that most of our processes reached level 5 (maximum of the scale) and need to be modified because management sensors, or customers vote that the process will not satisfy the requirements and expectations. Thus once again Process Mining can help to find here the process must be adapted.

Question:Can process mining be used in a continuous manner? Is it still interesting beyond initial use?

Answer: It can be used after change occurs and resume if changes were effective and the predicted outcomes were achieved. Is just like taking two CT scan, comparing if the cure eliminated the decease.

Question: I’d like to use process mining in order to find error causes in a manufacturing environment. The cause of these errors is often located in the details. How can you prevent abstraction away from  details too much when creating a process model?

Answer: With smart filtering it’s possible to look at the needle in the straw. I mean you can look at a particular category in a short period of time, look only to two activities in hundreds that occur in the process, look into two roles only …

Question: How is your experience to get the data you need to analyze processes? Do companies provide the data easily?

Answer: As data become an asset, its analysis and efficient exploration is important for companies to compete effectively. However the fragmentation and the volume of available data make it impossible to analyze manually. Today most of the enterprise systems have tools to extract data from databases, like knowledge views. I remember for example that BMPS databases are impossible to understand at extract data from it. Using tools like knowledge views are handy and easy to do it. SAP is known to have difficult to understand  database structure and we have experience extracting millions or records using ad-hoc query. One data attribute that is needed to process mining is the activity name. Activity names are not stored in systems with beautiful names like, created, progress, completed. Typically is stored state transition codes like 0, 10, 20, etc. If your IT dept. cannot provide help because there is no system documentation, initiate a new process instance and monitor how the state codes are recorded in the database and reverse engineering it.

One final thought: don’t let data greed to start playing a role in the project. Don’t extract data you don’t need to.

Question: Process Mining can analyze 100% of the cases, in an auditing context. How can you be sure that 30% of the cases are missing? Exactly the 30% that were fraudulent?

Answer: Some techniques can me use to cross check records. For example compare using a BI system the cases count with traces in the database, specially the ones that ere deleted.

Question: Can we apply the process mining techniques on a data of any Transaction Processing System? For example: on data generated by ATM?

Answer: Process mining can be used in transaction systems like the ones used by front office, when a customer pays a bill, request energy connection or pays goods at the supermarket.  It can show for example information about how the customers like to pay for transactions (cash, check, credit card) that combines with the process execution sequence that help managers to make decisions about customer preferences, or find causes that halts process execution (I just remember, transactions can take much more time at a given period of the day due communication infrastructure, and thus must be improved in order to not increase customer waiting time).

Question: What is the difference between the data mining and process mining.

Answer: Data mining is data-centric, not process oriented. I’m not saying data mining is useless. But does not show where and why organizations are for example performance lagging.  Business intelligence combined with data mining shows how many complaints were made, how many were answered and what response time, by category and other attributes (correlates important things like complains started by women take more time to handle, but if an offer is made to compensate the problem it’s more likely that the churn rate decreases)  but hides where people waste time or perform more tasks beyond that necessary, repetitions, duplication, etc. On the contrary, the process mining shows managers where they must act to achieve the performance expected and can act in real time, because they know in the part of the process that must be, for example accelerated or resolved.

Question: We have already implemented a BPM system. How does Process Mining fit to our BPM system?

Answer: BPMS record every interaction. Users log in and log out, every piece of data of all process instances, error logs and in some configurations used in military / defense or pharmaceutical industries every change that was made to data stored by the system (for example a user changed a test value from 4,0 to 4,3). Those interactions are recorded using what is called an audit trail (see image bellow). In this example you can see information about a process instance. By default it has all the data necessary to use process mining. Thus process mining tools integrate easily with BPMS.

BPMS Audit Trail

 

Business Analysis Conference Europe 2012 – closure

Here is a quick wrap up of day 2 (September 26th) rather than a long article. Day 0 can be found <here>, day one <here>.

Key note from David Clarke - Will the Business Change Profession Be up to the Challenge of the 5th Wave of Computing

Most of the times key notes are inspiring, but this one was for reflection.  When we ear Gartner and others presenting facts that are going to transform the world iInever believe them, not because I’m skeptic by the nature the work I do, but because they cannot be contrasted. For example, it’s being said that big data is the new information source to companies understand the world. I agree with that, but saying that zillions of data is being produced everyday, to justify IT investment and turn a decision maker into a person without strengths and ability to do new and improved decision making. But what is not justified is that most of that data is useless.

David talked about important things, curiously aligned with the presentation I gave at the BPM conference Europe 2012 and in the post BPM Agenda 2013.

He was telling that managers under their human condition are very bad predicting outcomes. He went to the story when he was working with Compaq computers and they were number 1 in the market, when Dell hit them. At the start the board did not believe that they could something, until they reinvented the business model (selling direct and customized computers), the rest is history … He picked that example to bring some light around of the characteristics of today’s business environment: volatility. Before companies sense new business models, most of them take months to replicate or even don’t have the capacity to do it and are are out of the market or lose a very important piece of the cake.

Them he emphasized the importance of human resource qualification and human soft skills. With the continuous erosion of outsourcing (I understood he was referring to the loss of Jaguar and Land Rover for example)  qualified people get out of the country that are necessary to handle the wave of technology disruption, meaning that without proper human resources, the country (he was referring to England but this applies to every other country) will lose ground and growth with impact in people’s life (jobs). Other interesting point of view was about people need to awake up to reality. Recession when over will not bring things back as they were. Companies do need to the adapt. Because when the storm is over it will even get worse (other will take your place).

David presented the shift from the operations driven organization (lets use the real name: process) to the people driven organization. It’s not outside-in and inside-out anymore. It’s a network of flows that some cannot be controlled with the bring your own device attitude and the copy paste to a way of work like we do in our private life. It’s the customer driving organization.

I loved the provocation about forget IT. IT is there to be used, don’t complicate, integrate.

Tamsin Fulton – Design Thinking to Refresh Business Analysis

Tamsim presented how a business analyst can create a common ground about a concept in a single sheet of of paper (rather is walls full of sticky notes). Making things visual and let people do some hands on with the parts of the process (using cards as parts of the process). Remembers some approaches from Professor Edward Tuffle. Her view is as valid as any other BPMN process model that’s is easy to understand.  I see Tamsin working also as a curator after a stretching sessions of endless process maps and make change happens to a new understandable vision that get’s buy in from the people. Then if it is necessary to translate the models for a workflow engine, for sure you can do it using other techniques.

Tony Buzan – Mind Maps for Business

I did not attend this session, but I’m a huge fan of mind maps. I leave here a comprehensive review from Penny Pullan (signed by Tony himself):

Penny Pullan

Some final remarks

Business Analysis as like other management domain cannot be scientifically validated. Thus there are lots of approaches, that can work and help to understand how organizations are structured and how people behave. Some analysis will end up in requirements engineering for software development,  but ultimately they must contribute to business change.

There was during the sessions and the during the breaks a lot of discussion about what a Business Analyst should do, to evolve or not to evolve  the practice, as also the typical turf wars between Analysts and IT and the way (I registered a lot of complains about this) how they are perceived by customers,  semantically speaking what is the difference between an internal consultant and an external analyst.

I’m tempted to say that like in other profession and in life Analysts are what they want to be. The trick is you become a linchpin during your path on earth. Finding new ways to think, to structure. Thus stand up a make a difference. Every day of your life.

Business Analysis Conference Europe 2012 – day 1

Here is a quick wrap up of day 1 (September 25th) rather than a long article. Day 0 can be found <here>. Closure can be found <here>.

Key note from Ben Goldrace

Ben provided some very interesting points about you as an Analyst gather evidence to support your analysis and decision. Some examples were about about scientific studies that are badly used to support reasoning. One the other of the points was people do not like to ear uncertainty. They wanted be correctly advised. Again this brings the importance of making the numbers based on facts not on assumptions.

Ben also talked about the importance of systematic review. It gave the example around medicine trials about giving steroids to pregnant woman to induce premature babies to born and reduce the probability the baby dies. Sadly, women will lose a baby during pregnancy or birth. He presented data about a study about field experimentation that if picked randomly like cherry picking,  pregnant (or other type of person) will not be able to make a decision. But if you put the information together it will help you to make the right choice. Now, this makes remember the hype created by Gartner about intelligent business process management. There is a difference that Gartner  must understand that is the line that divides of having data and a system that can makes recommendations by yourself. I feel that we will get there close, but I feel comfortable about how IBM’s Watson  answered about North Korea. It said : “the country does not have diplomatic relationship with the USA” .

To BA or not BA – is that the question?

by Lynda Girvan Debby Paul

This session and some side conversations during the breaks, brought to me the perspective that Business Analysts don’t need to go to the Psychiatrist. There is a mood among BA practitioners that they are considered as scribers by business people. Some argue that BA should be stand still inside of its aquarium, others say they need to evolve and touch other areas like solution implementation, process mapping and design, change management.

That was a surprise to me. But this is why knowledge areas continue to be siloed. Why BA don’t take the risk of every day when they wake up they try to be something different, rather than work on requirements engineering?

But surprisingly the audience at this session voted for the Analyst to continue to embrace the classical role, carry on the assessment with the output in mind  delivering requirement specification.  That’s lack of ambition I say, or maybe a cautious approach to avoid conflict (click to understand) like this one it emerged among two world recognized BPM experts that are discussing about who sells more books on Amazon.

Business Analysis Conference Europe 2012 – day 0

I’m attending at the Business Analysis Conference Europe and I will deliver a presentation about Process Mining. Day 1 coverage is <here>. Closure is <here>.

Here is a quick wrap up of day 0 (September 24th) rather than a long article.

Track 1 – Data modeling to represent business information needs – Keith Gordon

Keith talked about the importance of using models that does not take 6 months to draw and are difficult to understand by ordinary people. Despite the session was mostly about Entity Relationship diagrams (that in my humble opinion are less scary than UML class diagrams, can be scary to most business people) looks like it’s time to find other ways to describe organizations, domains, processes, because companies cannot wait so much time from change program kick off to implementation. Today on a Linkedin’s forum someone was commenting that is time to automate with native intelligence the way humans model concepts. Hence for those that want to take challenges its time to free our mind for thinking rather than drawing.

Lot’s of emphasis on the classic problem: business does not understand IT and vice versa, but this makes me think that:

Like any other knowledge domains, there are lots of methods, but all are wrong if they cannot align different tacit knowledge domains.

Also on teams, regarding the solo Analyst:

The single person modeler makes me remember the problem in human reality perception.

And on Analyst’s cognitive illusion, this example from Keith says it all:

What are the valid domain names for the gender? Make, Female? What abut In transition from male to female?

This one came from a chat during a break and is headed to BPM professionals:

The idea is BPM is complex and no one understands because professionals make it hard.

Track 2 – System thinking Emma Langman

Emma, take off with the importance of system thinking is about being fun and empathy. Rather than prescribing methods it dived around some useful approaches to understand concepts, like for example CATWOE defined by Peter Checkland as a part of his Soft Systems Methodology, or POSIWID from Stafford Beer on complex system analysis. I remember sometimes using CATWOE for stakeholders expectation assessment that is useful to reach to outcomes when implementing change. If you know from the start that a particular outcome is not going to see the light of the day, it’s time to start thinking how to tackle it, before strong opposition from a stakeholder group tackles you.

Another good point was:

If you want change happen, use simple language and metaphors.

The one above makes me remember the days I was in manufacturing and I need to listen to sport programs (that I hated) to proper communicate and joke with assembly line works to make things going.

There was one intriguing example about how the ideas and assumptions are formed. I was about two companies that produced pens. One factory produced blue pens and other red pens. The red pen factory had quality problems with the lid, the blue pen factory was one of a kind of a factory. Then the exercise was regarding creating ideas about the infrastructure and people’s culture for each company. Attendees painted a picture of the red pen factory as something filthy and dirty with screaming and desperate people. On the other hand, the blue factory was the perfect place to work.

I tend to admit that this exercise was more about prejudice and bias from the typical business analyst, because it’ possible (based on my experience) that most of the times you can have a shop floor you can kiss and that kind of company excels, but the contrary can also holds true (you can still kiss he floor but its not capable of doing things properly).

BPM Agenda 2013

We are somehow in the middle of a new transformation movement in the BPM arena, regarding the blend with social technologies. Social tech is responsible by the change we are engaging with others to work and opening new paths regarding approaches to collaborate and taking work execution out of enterprise boundaries. It’s funny how preachers and gurus one decade ago were trying to push the corporate world to design truly end to end processes and an unorganized movement (the social practice ) make a dream become true. I should say that this is a victory of the commons (or main street) that were able to show faster and intelligent collaboration was possible, contrary to some managers that pursue a rigid communication flow among people and peers (do not confuse with structured process approaches).

It is true that technology is introducing new possibilities (and I tend to think that the pervasiveness of new entrants like Internet of things will make change a step further) but on the other hand I feel that BPM is being driven by the available technology menu that halts managers to develop a vision how to continue to deliver value.

When we look back on the past achievements, he saw that business transformation as being publicize in case studies, BPM events and vendors solution portfolio, is focused on automation (sorry ACM partisans, but ACM is a new type of enterprise automation) and an important slice of the business is somehow being forgotten.

Challenges by industry sector:

Some industries examples bellow:

• Logistics – for logistics, what is important is maximizing the amount of cargo transported, optimizing itineraries, paperless transactions, location, seamless transport mode interchange, fraud detection and intelligent transport price calculation. Some of this challenges are being addressed by ERP (Hugh, ERP?) and custom system development, but most of the listed points are being ignored and diminishing the gross margin.
• Electrical energy – the goal of this type of company is sell energy, not reduce the time it takes to handle complains. For that it’s necessary to think and development “real” intelligent methods to maximize distribution lines availability and predictive failure models to make more with transformers and other type of equipment to produce, transport and distribute energy.
• Oil and Gas – there is a single and most important kpi in this type of industry. Making more of crude oil input that enters in refineries. The more you make from crude into diesel, gasoline and other oil products, the more you gain.
• Manufacturing – here, assembly line operations are key, together with product development innovation, logistics and order management. Maybe because manufacturing operations are continuously being “exported” to aspiring high GDP countries ( with consequences to the sovereign debt crises) are becoming out of the BPM event radar being put at the corner. For sure there are examples of technology that is helping people to communicate better and reduce errors (this is an area here we hear lots of time of substituting entirely the ERP by other types of BPM technology) nevertheless, improving manufacturing operations and deliver value added products it makes the difference. did you remember the Dell’s outcomes and ultimately Apple?

Thus, looks like the new wave of accomplishments will continually headed to the “new ” low hanging fruit, improve customer service, handle complex complains and requests, but the important challenges like the examples above is being forgotten. Why you don’t take a look into your value chain and start thinking to improve what really deliver value to your customers and where the money comes from?

Challenges by approach:

• Outsourcing: this is a zone where there are good and bad examples, but most of times it hurts the company. There are some goods approaches that lecture you to get rid of non added value processes. For example, if you are on manufacturing or retail, is better to get sell the trucks and lay off the truck drivers and the logistics personnel. The same applies for example if your company is a heavy manufacturing industry  and decide to outsource maintenance. The first 3 years are gold. Company balance sheet looks very good, the burden cost are wiped out, and the net results / earnings improve, but them it bounces back the other side of the coin. In Logistics outsourcing for example, service level degrades or if your partner has a highly structured service design, it will start charging you for forced stop time (when the truck arrives and the warehouse personnel does not want to load /unload it). In industrial Maintenance, the outsources does not had past experience dealing with the equipment (that knowledge was lost when you lay off your personnel) and down time increases, or unnecessary maintenance operations are executed, increasing maintenance cost. Solution for this? Maybe you should continue to design the process with your partner, or before go for outsource, radically improve it internally.

Challenges by design:

• Those who have read Steve Job’s biography, knows that he never admitted the possibility of using stylus pen for the iPhone, despite it make a lot easier to build the device. It’s curious that in the past, when I was in manufacturing I had budget restrictions and me and my team need to find radically new approaches to decrease the time to produce and improve product quality. The stylus story makes me remember when it was necessary to improve the freezer evaporator. The current approach was to change the design of the coils that would induce more men / labor or invest in a automatic coiling machine (but I had no budget). On the other hand coil evaporator had sometimes problems with gas leaks, that make it much less reliable. I put on the top of the table to change the evaporator system to a single aluminum sheet. In those days, that were not the industry standard and it cost a lot of money because there were not scale economies. I worked close with supplier and make it happen. In the end a better product was delivered to the market, it took less time to produce and contribute to increase the margin. When was the last time you think it’s time to bring innovation by design to your products and services?

 

The challenge, help to construct the agenda:

On purpose I do not want to extend the post to every possible scenario (also because its a sign of lack of humility). Thus I leave where a challenge: complete the agenda and leave a comment. Who knows this can become a program for an upcoming independent BPM event.

The three layers of social business – preview

 

Note: This post does not reflect anymore my point of view on this subject. For historical purposes I keep record of this post. To access the new vision on Social Business, please access  A Social Platform Definition and related content from there.

 

This is a mirror post from eBizQ post before enters into oblivion. Somehow it shows one of the areas I’m working with some companies regarding the three layers for social interaction. Social execution is not about having a social tech pallet or stack as most of analysts like to call.

Social interaction is still in early adoption steps. Adoption will increase as managers realize that customers are taking control of process execution. Those that does not make the sift will suffer from poor customer experience and will harm business.

There are two driving factors that are making the change.

1- The type of company value chain. If the company sells / provides directly to consumers, means that they cannot control the touching points with the company (consumers can now express freely frustration, disappointment, change the execution path because they have mobile ammunition on the go to access and send information).

2 – Social technology. People are getting used to replicate communication habits with every entity they contact. This means they will or are keen of exchange information with companies they buy products, what information, support, whatever.

But this not enough. Again we read / listen all the time about social technologies that will make the change. On one hand is true that tech increases information flow speed making the organization more horizontal, improving communication, information exchange, building better, more participative teams; on the other hand you need alignment between these three layers:

1- The social practice, or the business process, instead of thinking about the typical process flow with activities, that don’t have the granularity and the detail to express the dynamics of social interaction, think more about a conversation to achieve a process goal. That conversation can be carried using different, mixed technologies, voice, text or video. Conversations involve people that belong to network nodes and conversation occurs within the arcs of the network.

2- The organization layer, not the organizational structure with business units and departments and roles, but the social network, that shows the communication flow and how people are connected with.

3 – The Technology layer, the applications people use to communicate and socialize. Some examples are activity streams, e-mail, wikis, instant messaging, whatever it fits their needs. It can happen, that different people in different organizations communicate the same piece of information using different tools, because they tend to adapt to its needs or regarding IT rules (for example IT Dept, don’t like to allow Google docs, because of its lack of privacy and security concerns).

Thus social tech is not only what it counts. Putting all the three layers is what makes the difference.

I would say that the company social network patterns will make the difference regarding the type of knowledge that is applied. Something a got deeper in this blog post.

On Process maturity assessments

Process maturity models start from the assumption that such a model is capable to identify process performance from multiple perspectives and setup an enterprise change program.

Dan Kane‘s post is clear regarding pure capabilities of this advanced models.

A typical maturity assessment starts with the assumption, implicit or explicit, that improved process maturity is an end goal in and of itself.  Here is a great example of what I mean, based on the CMMI model of evaluating maturity.  The definition of Level 1 maturity for Incident Management indicates there is no defined owner of the process.  Part of reaching Level 2 maturity is identifying a single process owner.  I agree that having a single owner of the process is a good thing, but what goal does it help achieve?  Moving to Maturity Level 2?  Congratulations.  My concern is that many maturity assessments end there.  You’re at Level 2 (or 4, it doesn’t matter).  Now what?

One month ago Capgemini published a new BPM report state of the nation. This study show evidence that many companies reported that most of its business process had a maturity level of 3 : defined and compliant with business rules (or any similar outcome) and what to move to level 4 (in a scale of 5).

During this year enterprises I work with are facing continuum pressure to improve operational processes, focused on cutting costs. Change candidates are normally the ones that use more resources. Typically is being said maturity models help managers to make decisions where to start change programs, but practice shows that managers ignore it. This is because maturity models ignore proper human reasoning and human perception of what must be changed. Managers feel correctly what is needed to be transformed.

Do you believe it’s worth to start a change program because you have a process at level 3 maturity stage when a level 5 process, customer critical is the first candidate? What if all processes equalize at level 5 (like human assessment methods that after 3 years of implementation 80% of employees reach the top of the scale).

Hence, what is the value of maturity models?