tag:www.mixpanelstatus.com,2005:/historyMixpanel Status - Incident History2024-03-29T00:52:53-07:00Mixpaneltag:www.mixpanelstatus.com,2005:Incident/203330872024-03-22T17:33:55-07:002024-03-22T17:33:55-07:00Degraded Query API performance impacting US reports<p><small>Mar <var data-var='date'>22</var>, <var data-var='time'>17:33</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Mar <var data-var='date'>22</var>, <var data-var='time'>16:56</var> PDT</small><br><strong>Monitoring</strong> - A fix was implemented, and reports should now be loading. We are monitoring the results.</p><p><small>Mar <var data-var='date'>22</var>, <var data-var='time'>16:24</var> PDT</small><br><strong>Identified</strong> - The issue has been identified, and a fix was implemented.</p><p><small>Mar <var data-var='date'>22</var>, <var data-var='time'>16:13</var> PDT</small><br><strong>Investigating</strong> - Mixpanel is experiencing degraded Query API performance, causing latency or timeouts when loading data in reports. This is currently affecting US projects.<br /><br />We truly appreciate your patience and apologize for the inconvenience. If you have any questions, please <a href="https://mixpanel.com/get-support">contact support</a></p>tag:www.mixpanelstatus.com,2005:Incident/201860312024-03-07T19:28:16-08:002024-03-07T19:28:16-08:00People pipeline delays and tasks stuck<p><small>Mar <var data-var='date'> 7</var>, <var data-var='time'>19:28</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Mar <var data-var='date'> 7</var>, <var data-var='time'>13:48</var> PST</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Mar <var data-var='date'> 7</var>, <var data-var='time'>07:52</var> PST</small><br><strong>Investigating</strong> - We're currently experiencing delays impacting some projects with people pipelines (export). Our Engineering team on call has been alerted and is initiating an investigation to identify and resolve the issue.<br /><br />We apologize for any inconvenience this may cause and appreciate your patience as we work diligently to restore normal operations. We'll provide updates as soon as more information becomes available here.</p>tag:www.mixpanelstatus.com,2005:Incident/201606322024-03-04T17:17:47-08:002024-03-04T17:17:47-08:00Temporary high latency with the /import API<p><small>Mar <var data-var='date'> 4</var>, <var data-var='time'>17:17</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Mar <var data-var='date'> 4</var>, <var data-var='time'>14:29</var> PST</small><br><strong>Update</strong> - We are continuing to monitor for any further issues.</p><p><small>Mar <var data-var='date'> 4</var>, <var data-var='time'>14:29</var> PST</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Mar <var data-var='date'> 4</var>, <var data-var='time'>13:35</var> PST</small><br><strong>Investigating</strong> - We're experiencing an elevated level of API errors for the /import API using service account authentication and are currently looking into the issue.</p>tag:www.mixpanelstatus.com,2005:Incident/201017282024-02-28T18:55:05-08:002024-02-28T18:55:05-08:00Raw Event Export API degraded for a subset of projects<p><small>Feb <var data-var='date'>28</var>, <var data-var='time'>18:55</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Feb <var data-var='date'>28</var>, <var data-var='time'>15:26</var> PST</small><br><strong>Investigating</strong> - A subset of projects are experiencing issues exporting data through our raw export API. Our engineering team is currently investing and we will provide an update as soon as possible. Event ingestion is not affected.<br /><br />We appreciate your patience while our engineers work to restore timely service. If you have any questions, please <a href="https://mixpanel.com/get-support">contact support</a></p>tag:www.mixpanelstatus.com,2005:Incident/199715392024-02-12T20:28:00-08:002024-02-12T20:28:00-08:00Temporary data ingestion delay<p><small>Feb <var data-var='date'>12</var>, <var data-var='time'>20:28</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Feb <var data-var='date'>12</var>, <var data-var='time'>20:02</var> PST</small><br><strong>Update</strong> - We are continuing to monitor for any further issues.</p><p><small>Feb <var data-var='date'>12</var>, <var data-var='time'>20:02</var> PST</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Feb <var data-var='date'>12</var>, <var data-var='time'>18:25</var> PST</small><br><strong>Identified</strong> - The issue has been identified, and a fix is being implemented.</p><p><small>Feb <var data-var='date'>12</var>, <var data-var='time'>18:24</var> PST</small><br><strong>Investigating</strong> - We are experiencing delays with our data ingestion pipeline for a small subset of US projects. As a result, real-time data for a subset of projects is delayed. No data is being lost. We appreciate your patience while our engineers work to restore real-time functionality. If you have any questions, please <a href="https://mixpanel.com/get-support">contact support</a></p>tag:www.mixpanelstatus.com,2005:Incident/199386312024-02-07T23:00:00-08:002024-02-08T03:52:43-08:00Temporary high latency with the /import API<p><small>Feb <var data-var='date'> 7</var>, <var data-var='time'>23:00</var> PST</small><br><strong>Resolved</strong> - We experienced a temporary high latency issue affecting the /import API. Our Engineering Team on call was alerted and resolved the issue. The incident lasted from 7 AM UTC to 9 AM UTC. The incident is now resolved after monitoring the API and verified there is no latency and all systems are operational.</p>tag:www.mixpanelstatus.com,2005:Incident/199337262024-02-07T14:31:57-08:002024-02-07T14:31:57-08:00Temporary data ingestion delay<p><small>Feb <var data-var='date'> 7</var>, <var data-var='time'>14:31</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Feb <var data-var='date'> 7</var>, <var data-var='time'>14:26</var> PST</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Feb <var data-var='date'> 7</var>, <var data-var='time'>13:30</var> PST</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Feb <var data-var='date'> 7</var>, <var data-var='time'>12:40</var> PST</small><br><strong>Investigating</strong> - We are experiencing delays with our data ingestion pipeline. As a result, data from 6am PST onward may be delayed for a subset of projects. No data is being lost. We appreciate your patience while our engineers work to restore functionality. If you have any questions, please <a href="https://mixpanel.com/get-support">contact support</a>.</p>tag:www.mixpanelstatus.com,2005:Incident/199325792024-02-07T12:51:39-08:002024-02-07T12:51:39-08:00Degraded API Performance Impacting Reports and Cohort Syncs for a Subset of Projects<p><small>Feb <var data-var='date'> 7</var>, <var data-var='time'>12:51</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Feb <var data-var='date'> 7</var>, <var data-var='time'>10:41</var> PST</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Feb <var data-var='date'> 7</var>, <var data-var='time'>09:56</var> PST</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented. <br /><br />A subset of customers are experiencing an increased error rate for our /engage endpoint. This is impacting reports leveraging cohorts which query date ranges larger than 40 days. This has also impacted cohort sync integrations.</p><p><small>Feb <var data-var='date'> 7</var>, <var data-var='time'>09:52</var> PST</small><br><strong>Investigating</strong> - A subset of projects are experiencing issues loading reports utilizing cohort analysis and an increased error rate for cohort sync integrations. Our engineers are working to fix this issue as quickly as possible. We truly appreciate your patience and apologize for the inconvenience. If you have any questions, please <a href="https://mixpanel.com/get-support">contact support</a>.</p>tag:www.mixpanelstatus.com,2005:Incident/199254462024-02-06T16:06:01-08:002024-02-06T16:06:01-08:00Elevated API Errors<p><small>Feb <var data-var='date'> 6</var>, <var data-var='time'>16:06</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Feb <var data-var='date'> 6</var>, <var data-var='time'>14:19</var> PST</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Feb <var data-var='date'> 6</var>, <var data-var='time'>13:59</var> PST</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Feb <var data-var='date'> 6</var>, <var data-var='time'>13:50</var> PST</small><br><strong>Investigating</strong> - We're experiencing an elevated level of API errors for the /import API using service account authentication and are currently looking into the issue.</p>tag:www.mixpanelstatus.com,2005:Incident/199112282024-02-06T13:54:51-08:002024-02-06T13:54:51-08:00EU Pipeline Export Delays<p><small>Feb <var data-var='date'> 6</var>, <var data-var='time'>13:54</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Feb <var data-var='date'> 6</var>, <var data-var='time'>07:11</var> PST</small><br><strong>Monitoring</strong> - The recent delays experienced with Export Pipelines in EU have been addressed by the previous fix. Our Engineering team have diligently worked to rectify the issue, and services are now operating as expected. We will publish an update here when the issue is considered fully resolved. As of now, our monitoring systems indicate that all services are functioning normally. However, our on-call team will continue to closely monitor the situation to ensure uninterrupted service delivery within our export SLA.<br /><br />We apologize for any inconvenience this incident may have caused and appreciate your patience and understanding as we worked to resolve it. Should you encounter any further issues or have any concerns, please do not hesitate to reach out to our Support team.</p><p><small>Feb <var data-var='date'> 5</var>, <var data-var='time'>21:01</var> PST</small><br><strong>Update</strong> - We are continuing to work on a fix for this issue.</p><p><small>Feb <var data-var='date'> 5</var>, <var data-var='time'>09:23</var> PST</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Feb <var data-var='date'> 5</var>, <var data-var='time'>02:24</var> PST</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Feb <var data-var='date'> 5</var>, <var data-var='time'>02:24</var> PST</small><br><strong>Investigating</strong> - We are currently experiencing delays in exports setup through our Pipelines functionality that is affecting some of our EU customers. Thank you for your patience as the Engineering team investigates this. If you have additional questions, please reach out to <a href="https://help.mixpanel.com/hc/en-us/requests/new">support</a>.</p>tag:www.mixpanelstatus.com,2005:Incident/199162642024-02-05T11:30:00-08:002024-02-05T13:31:41-08:00US Pipeline Export Delays<p><small>Feb <var data-var='date'> 5</var>, <var data-var='time'>11:30</var> PST</small><br><strong>Resolved</strong> - We identified a Data Pipeline export delay affecting some US-based projects. The delay led to some hourly exports taking longer to complete. A fix has been released for hourly exports going forward. Past data will be fixed when sync runs for the pipeline.</p>tag:www.mixpanelstatus.com,2005:Incident/198893712024-02-01T17:20:31-08:002024-02-01T18:07:13-08:00Degraded WebApp Performance<p><small>Feb <var data-var='date'> 1</var>, <var data-var='time'>17:20</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Feb <var data-var='date'> 1</var>, <var data-var='time'>17:11</var> PST</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Feb <var data-var='date'> 1</var>, <var data-var='time'>16:24</var> PST</small><br><strong>Investigating</strong> - We are experiencing degraded performance loading pages in our WebApp. <br />We appreciate your patience while our engineers work to restore normal functionality. If you have any questions, please <a href="https://mixpanel.com/get-support">contact support</a></p>tag:www.mixpanelstatus.com,2005:Incident/198829062024-02-01T08:56:07-08:002024-02-01T08:56:07-08:00Degraded API Performance Impacting Reports for EU projects<p><small>Feb <var data-var='date'> 1</var>, <var data-var='time'>08:56</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Feb <var data-var='date'> 1</var>, <var data-var='time'>01:51</var> PST</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Feb <var data-var='date'> 1</var>, <var data-var='time'>01:10</var> PST</small><br><strong>Investigating</strong> - Projects in our EU cluster are experiencing issues loading reports and exporting data through the API endpoint mixpanel.com/api/2.0/. Our engineers are working to fix this issue as quickly as possible. <br /><br />We truly appreciate your patience and apologize for the inconvenience.</p>tag:www.mixpanelstatus.com,2005:Incident/198056212024-01-23T11:44:01-08:002024-01-23T11:44:01-08:00Elevated latency detected in the /import endpoint<p><small>Jan <var data-var='date'>23</var>, <var data-var='time'>11:44</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jan <var data-var='date'>23</var>, <var data-var='time'>10:34</var> PST</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Jan <var data-var='date'>23</var>, <var data-var='time'>07:02</var> PST</small><br><strong>Update</strong> - our engineering team continues to investigate the issue with higher latencies. Thanks so much for your patience while we review this</p><p><small>Jan <var data-var='date'>23</var>, <var data-var='time'>04:47</var> PST</small><br><strong>Investigating</strong> - Our team has recently been alerted that there's an increase in latency for our /import endpoint for data ingestion, which can slow down sending data to Mixpanel in large volumes. No data is being lost . We appreciate your patience while our engineers investigate the situation. If you have any questions, please <a href="https://mixpanel.com/get-support">contact support</a></p>tag:www.mixpanelstatus.com,2005:Incident/196906422024-01-11T16:00:00-08:002024-01-12T10:30:22-08:00Temporary ID Merge Delay<p><small>Jan <var data-var='date'>11</var>, <var data-var='time'>16:00</var> PST</small><br><strong>Resolved</strong> - We identified a delay with our identity management system around Jan 11 ~4 PM PST where it took more than 24 hours for identity management mappings to reflect their merged state in reports. As a result, reports such as Funnels may have shown a decrease in conversion. The delay has since been resolved, and merges should now propagate in reports within 24 hours.</p>tag:www.mixpanelstatus.com,2005:Incident/196712962024-01-10T17:39:31-08:002024-01-10T17:39:34-08:00Temporary ID Merge and Ingestion Delay<p><small>Jan <var data-var='date'>10</var>, <var data-var='time'>17:39</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Jan <var data-var='date'>10</var>, <var data-var='time'>15:51</var> PST</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Jan <var data-var='date'>10</var>, <var data-var='time'>15:09</var> PST</small><br><strong>Identified</strong> - The ID Merge delay mentioned in this incident has now been resolved. <br />However, there is currently an ingestion delay related to this incident. As a result, real-time event data is delayed. No data is being lost.<br />A fix has been implemented for the ingestion delay, and we will continue to provide additional updates on this incident.</p><p><small>Jan <var data-var='date'>10</var>, <var data-var='time'>12:48</var> PST</small><br><strong>Investigating</strong> - We are experiencing delays with our identity management system.<br />As a result, ID Merge and $identify, $merge, and $create_alias events are delayed.<br />No data is being lost. We appreciate your patience while our engineers work to restore real-time functionality. If you have any questions, please contact support.</p>tag:www.mixpanelstatus.com,2005:Incident/193579812023-12-06T17:01:59-08:002023-12-06T17:01:59-08:00Degraded API Performance Impacting Reports for a Subset of U.S. Projects<p><small>Dec <var data-var='date'> 6</var>, <var data-var='time'>17:01</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Dec <var data-var='date'> 6</var>, <var data-var='time'>14:35</var> PST</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Dec <var data-var='date'> 6</var>, <var data-var='time'>14:01</var> PST</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Dec <var data-var='date'> 6</var>, <var data-var='time'>13:20</var> PST</small><br><strong>Update</strong> - We are continuing to investigate this issue.</p><p><small>Dec <var data-var='date'> 6</var>, <var data-var='time'>12:58</var> PST</small><br><strong>Investigating</strong> - A subset of U.S. projects are experiencing issues loading reports. Our engineers are working to fix this issue as quickly as possible. We truly appreciate your patience and apologize for the inconvenience. If you have any questions, please contact Support at support@mixpanel.com.</p>tag:www.mixpanelstatus.com,2005:Incident/192566662023-12-01T08:51:18-08:002023-12-01T08:51:18-08:00Degraded API Performance Impacting Reports for a Subset of Projects<p><small>Dec <var data-var='date'> 1</var>, <var data-var='time'>08:51</var> PST</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Dec <var data-var='date'> 1</var>, <var data-var='time'>04:38</var> PST</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Dec <var data-var='date'> 1</var>, <var data-var='time'>00:19</var> PST</small><br><strong>Identified</strong> - The engineering team has issued a fix and reports are loading for most of the affected customers. The team is still working on reviewing the situation for a small subset of the affected customers.</p><p><small>Nov <var data-var='date'>30</var>, <var data-var='time'>23:15</var> PST</small><br><strong>Investigating</strong> - A subset of projects are experiencing issues loading reports and exporting data through the API endpoint mixpanel.com/api/2.0/. Our engineers are working to fix this issue as quickly as possible. We truly appreciate your patience and apologize for the inconvenience. If you have any questions, please <a href="https://mixpanel.com/get-support">contact support</a>.</p>tag:www.mixpanelstatus.com,2005:Incident/190085892023-11-03T16:13:08-07:002023-11-03T16:13:21-07:00/import endpoint Data Ingestion Delay<p><small>Nov <var data-var='date'> 3</var>, <var data-var='time'>16:13</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Nov <var data-var='date'> 3</var>, <var data-var='time'>12:17</var> PDT</small><br><strong>Update</strong> - The real-time functionality of /import has now been restored for most projects. We are continuing to monitor for any further issues.</p><p><small>Nov <var data-var='date'> 3</var>, <var data-var='time'>10:55</var> PDT</small><br><strong>Update</strong> - We are continuing to monitor while the enqueued events are being processed.<br />Our current estimation to process the backlog of data is around ~2 hours for most projects before real-time functionality of /import is restored.<br />We will continue to provide additional updates on the situation as new information becomes available.</p><p><small>Nov <var data-var='date'> 3</var>, <var data-var='time'>06:04</var> PDT</small><br><strong>Update</strong> - We are continuing to monitor while the enqueued events are processed. Given the substantial backlog, we anticipate that this processing may require a considerable amount of time to complete. Although we are unable to provide an ETA at this moment, we will continue to update this incident as soon as new information becomes available.</p><p><small>Nov <var data-var='date'> 3</var>, <var data-var='time'>03:10</var> PDT</small><br><strong>Monitoring</strong> - We have pinpointed the problem responsible for the delays and have implemented a solution. We are currently monitoring the result, we will provide additional updates on the situation once it is resolved.</p><p><small>Nov <var data-var='date'> 3</var>, <var data-var='time'>02:23</var> PDT</small><br><strong>Identified</strong> - We are experiencing delays with our event data ingestion pipeline. As a result, real-time event data sent through /import is delayed. <br />This incident also affects any integrations sending data through /import. <br />We appreciate your patience while our engineers work to restore real-time functionality. If you have any questions, please <a href="https://mixpanel.com/get-support">contact support</a></p>tag:www.mixpanelstatus.com,2005:Incident/189736762023-10-30T16:30:00-07:002023-10-30T17:24:41-07:00Degraded Query API for a subset of projects affecting reports<p><small>Oct <var data-var='date'>30</var>, <var data-var='time'>16:30</var> PDT</small><br><strong>Resolved</strong> - A subset of projects experienced Query API degradation, causing issues loading reports and exporting data through the query API endpoint. The Engineering team deployed a fix to address the issue on October 30th at ~4:50 pm PST. Query requests are now returning data normally.<br /><br />If you have any questions, please <a href="https://mixpanel.com/get-support">contact support</a></p>tag:www.mixpanelstatus.com,2005:Incident/189071842023-10-23T17:00:00-07:002023-10-23T20:19:21-07:00504 errors for /import and other APIs that required Auth Service Accounts<p><small>Oct <var data-var='date'>23</var>, <var data-var='time'>17:00</var> PDT</small><br><strong>Resolved</strong> - We had an incident where /import and other APIs that use Auth Service Accounts were running into 504 errors. This incident started at at 4:58pm PT and was fixed by 7:44pm PT. If you experienced any errors during this time, we ask that you replay the events that were sent with /import.</p>tag:www.mixpanelstatus.com,2005:Incident/188172412023-10-21T10:17:25-07:002023-10-21T10:17:25-07:00Merge is currently delayed for a subset of projects<p><small>Oct <var data-var='date'>21</var>, <var data-var='time'>10:17</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Oct <var data-var='date'>21</var>, <var data-var='time'>08:07</var> PDT</small><br><strong>Update</strong> - The backlog of data has been processed and delays should have returned to normal levels. We are currently monitoring to ensure all projects have fully recovered.</p><p><small>Oct <var data-var='date'>20</var>, <var data-var='time'>17:39</var> PDT</small><br><strong>Update</strong> - Our Engineering team remains at work continuing to process the backlog of data. Our current estimation for processing the remaining backlog is Sunday, October 22 by end of day PT time.<br /><br />We greatly appreciate your patience as we continue to work diligently on resolving this issue. We'll keep you updated here as we make progress.</p><p><small>Oct <var data-var='date'>19</var>, <var data-var='time'>21:57</var> PDT</small><br><strong>Update</strong> - Our Engineering team on call continues to process the backlog of data. At this time, projects with IDs below 2500000 may be intermittently seeing accurate data from Oct 10th through the 17th. There is no data loss and events will show in reports as expected when the backlog finishes processing.<br /><br />We will continue to update as we get more information, thank you for your patience.</p><p><small>Oct <var data-var='date'>19</var>, <var data-var='time'>08:42</var> PDT</small><br><strong>Update</strong> - At this time, the Engineering team remains at work addressing the backlog with unifying data from merged IDs. Our estimation is that merging times will be back to normal by Friday the 20th by end of day PT time. For events already merged, but not showing, per the previous update, this is limited to a subset of events from Oct 10th through the 17th. Those will show in reports normally as the backlog finishes processing on Friday. Events after the 17th are not affected.<br /><br />We greatly appreciate your patience as we continue to work diligently on resolving this issue. We'll keep you updated here as we make progress.</p><p><small>Oct <var data-var='date'>18</var>, <var data-var='time'>18:37</var> PDT</small><br><strong>Update</strong> - The Engineering team continues to process the backlog of data, and our estimate to process the backlog continues to stay on track. We are currently estimating ~40 hours to return to normal. <br /><br />We discovered is that some data from October 10-17th is not appearing in reports for some projects. There is no data loss and events will re-appear in your project as expected when the incident is resolved. <br /><br />Thank you for your patience while we work on resolving this.</p><p><small>Oct <var data-var='date'>18</var>, <var data-var='time'>08:26</var> PDT</small><br><strong>Update</strong> - We'd like to provide a brief update on the estimated resolution time for the current data processing backlog. Our current estimate is that it will take approximately 48 hours to process the backlog and return to normal.<br /><br />We appreciate your patience during this time, and we're committed to getting things back on track as quickly as possible. Rest assured, we're diligently working on resolving this matter.</p><p><small>Oct <var data-var='date'>18</var>, <var data-var='time'>05:40</var> PDT</small><br><strong>Update</strong> - Our Engineering team on call has been working tirelessly to address the backlog of pending merge operations, and we appreciate your patience as we continue to make progress on this front. While we cannot provide an exact timeline, we estimate that it may take another 24 hours to fully clear the queue and fully resolve the issue for all projects but we will provide a more accurate resolution time in the next update. We understand that this situation has affected data reporting, which may have led to a higher drop-off in funnels for the last week and the potential for some events to be missing from your reports. We want to reassure you that no data has been lost, and we are actively working on processing all data to ensure the integrity and completeness of your reports.<br />Our team is committed to resolving this matter as swiftly as possible. We will provide further updates as we make progress and expect to give you a final resolution notice once the incident is fully resolved. We sincerely apologize for any inconvenience this may have caused and appreciate your understanding as we work to resolve this issue.</p><p><small>Oct <var data-var='date'>17</var>, <var data-var='time'>17:29</var> PDT</small><br><strong>Update</strong> - We are continuing to monitor for any further issues.</p><p><small>Oct <var data-var='date'>17</var>, <var data-var='time'>09:33</var> PDT</small><br><strong>Update</strong> - We are continuing to monitor for any further issues.</p><p><small>Oct <var data-var='date'>17</var>, <var data-var='time'>00:41</var> PDT</small><br><strong>Update</strong> - A fix has been deployed to resolve this issue. However, it is important to note that the fix is expected to take a minimum of 24 hours to complete due to the current backlog of merge operations. We are committed to reducing this time as much as possible while ensuring a thorough resolution. The Engineering team on call is actively monitoring the situation round the clock to ensure the fix progresses smoothly. We will continue to provide updates as we make progress and expect to give you a final resolution notice once the incident is fully resolved.</p><p><small>Oct <var data-var='date'>16</var>, <var data-var='time'>15:40</var> PDT</small><br><strong>Monitoring</strong> - A fix has been deployed, which will take 24+ hours to resolve the delayed merges. We are monitoring the results while the fix is in progress.</p><p><small>Oct <var data-var='date'>16</var>, <var data-var='time'>13:44</var> PDT</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Oct <var data-var='date'>16</var>, <var data-var='time'>11:44</var> PDT</small><br><strong>Investigating</strong> - A subset of projects is currently experiencing a delay with merging profiles and event data since October 10th. We appreciate your patience while our engineers work to restore real-time functionality. If you have any questions, please contact support.</p>tag:www.mixpanelstatus.com,2005:Incident/186670652023-10-02T17:00:44-07:002023-10-03T01:42:12-07:00Elevated API errors for the raw export endpoint<p><small>Oct <var data-var='date'> 2</var>, <var data-var='time'>17:00</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Oct <var data-var='date'> 2</var>, <var data-var='time'>10:50</var> PDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Oct <var data-var='date'> 2</var>, <var data-var='time'>09:08</var> PDT</small><br><strong>Update</strong> - Our engineering team continues to investigate the issue. Thanks for your patience.</p><p><small>Oct <var data-var='date'> 2</var>, <var data-var='time'>07:32</var> PDT</small><br><strong>Investigating</strong> - We're experiencing an elevated level of errors in our API to export raw event data. Our engineering team has been alerted and is in the process of investigating. If you have any questions, please <a href="https://mixpanel.com/get-support">contact support</a></p>tag:www.mixpanelstatus.com,2005:Incident/185597762023-09-20T16:36:21-07:002023-09-20T16:36:21-07:00Errors loading Mixpanel.com<p><small>Sep <var data-var='date'>20</var>, <var data-var='time'>16:36</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Sep <var data-var='date'>20</var>, <var data-var='time'>15:31</var> PDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Sep <var data-var='date'>20</var>, <var data-var='time'>14:55</var> PDT</small><br><strong>Identified</strong> - The issue has been identified and a fix is being implemented.</p><p><small>Sep <var data-var='date'>20</var>, <var data-var='time'>14:09</var> PDT</small><br><strong>Investigating</strong> - We have identified issues loading mixpanel.com/eu.mixpanel.com, and elevated errors from Raw Export API. We are working to restore service as quickly as possible.</p>tag:www.mixpanelstatus.com,2005:Incident/185539202023-09-20T09:39:46-07:002023-09-20T09:39:46-07:00Elevated errors loading report for projects in the EU<p><small>Sep <var data-var='date'>20</var>, <var data-var='time'>09:39</var> PDT</small><br><strong>Resolved</strong> - This incident has been resolved.</p><p><small>Sep <var data-var='date'>20</var>, <var data-var='time'>08:44</var> PDT</small><br><strong>Update</strong> - A fix was successfully deployed a few hours ago, and it appears that no further issues have arisen. Nevertheless, we are continuing to closely monitor the situation as a precautionary measure.</p><p><small>Sep <var data-var='date'>20</var>, <var data-var='time'>03:02</var> PDT</small><br><strong>Monitoring</strong> - A fix has been implemented and we are monitoring the results.</p><p><small>Sep <var data-var='date'>20</var>, <var data-var='time'>02:12</var> PDT</small><br><strong>Investigating</strong> - We are currently investigating an issue where reports for projects in the EU cluster are not loading.<br />We truly appreciate your patience and apologize for the inconvenience. If you have any questions, please <a href="https://mixpanel.com/get-support">contact support</a></p>