Incidents reported on status page for Vanillo https://status.vanillo.tv/ https://d1lppblt9t2x15.cloudfront.net/logos/52f8b71f4671c9566edf0f51bea6489a.png https://status.vanillo.tv/ en https://status.vanillo.tv/ Sun, 25 May 2025 23:10:34 +0000 https://status.vanillo.tv/#d80022e93cd22c5bcb8fa42527167b4932f74aefc496dcf6e39f4c9c5bccb00c Vanillo API recovered https://status.vanillo.tv/ Sun, 25 May 2025 23:07:38 +0000 https://status.vanillo.tv/#d80022e93cd22c5bcb8fa42527167b4932f74aefc496dcf6e39f4c9c5bccb00c Vanillo API went down https://status.vanillo.tv/incident/566725 Sun, 25 May 2025 21:17:00 -0000 https://status.vanillo.tv/incident/566725#66e15bac53859df8bb3a3586a5399536a195b643bee17ed1d4a34e6b81f8731c Vanillo is back online! We sincerely apologize for the extended downtime and appreciate your patience. Please let us know if you encounter any issues. https://status.vanillo.tv/incident/566725 Sun, 25 May 2025 21:17:00 -0000 https://status.vanillo.tv/incident/566725#66e15bac53859df8bb3a3586a5399536a195b643bee17ed1d4a34e6b81f8731c Vanillo is back online! We sincerely apologize for the extended downtime and appreciate your patience. Please let us know if you encounter any issues. https://status.vanillo.tv/incident/566725 Sun, 25 May 2025 21:17:00 -0000 https://status.vanillo.tv/incident/566725#66e15bac53859df8bb3a3586a5399536a195b643bee17ed1d4a34e6b81f8731c Vanillo is back online! We sincerely apologize for the extended downtime and appreciate your patience. Please let us know if you encounter any issues. https://status.vanillo.tv/incident/566725 Sun, 25 May 2025 21:17:00 -0000 https://status.vanillo.tv/incident/566725#66e15bac53859df8bb3a3586a5399536a195b643bee17ed1d4a34e6b81f8731c Vanillo is back online! We sincerely apologize for the extended downtime and appreciate your patience. Please let us know if you encounter any issues. https://status.vanillo.tv/incident/566725 Sun, 25 May 2025 21:17:00 -0000 https://status.vanillo.tv/incident/566725#66e15bac53859df8bb3a3586a5399536a195b643bee17ed1d4a34e6b81f8731c Vanillo is back online! We sincerely apologize for the extended downtime and appreciate your patience. Please let us know if you encounter any issues. https://status.vanillo.tv/ Sun, 25 May 2025 20:52:36 +0000 https://status.vanillo.tv/#724602fb7b903febf5498c2f8fce48768fff1af387571baef50702c8fbc840e0 Vanillo API recovered https://status.vanillo.tv/ Sun, 25 May 2025 20:49:35 +0000 https://status.vanillo.tv/#724602fb7b903febf5498c2f8fce48768fff1af387571baef50702c8fbc840e0 Vanillo API went down https://status.vanillo.tv/ Sun, 25 May 2025 19:19:31 +0000 https://status.vanillo.tv/#107cbfc4ddf2510fdfa8f9015526a547e77b49482f364331090ba67486d47f13 Vanillo API recovered https://status.vanillo.tv/ Sun, 25 May 2025 19:04:32 +0000 https://status.vanillo.tv/#107cbfc4ddf2510fdfa8f9015526a547e77b49482f364331090ba67486d47f13 Vanillo API went down https://status.vanillo.tv/incident/566725 Fri, 16 May 2025 05:18:00 -0000 https://status.vanillo.tv/incident/566725#f94e48822ec1fb4ea0d01263d871d0a2a5a892a7be6637f8b21de6177c3e60df Due to an unforseen issue in the cluster maintenence, we are going to be placing the site into maintenance mode indefinitely to work out a fix. https://status.vanillo.tv/incident/566725 Fri, 16 May 2025 05:18:00 -0000 https://status.vanillo.tv/incident/566725#f94e48822ec1fb4ea0d01263d871d0a2a5a892a7be6637f8b21de6177c3e60df Due to an unforseen issue in the cluster maintenence, we are going to be placing the site into maintenance mode indefinitely to work out a fix. https://status.vanillo.tv/incident/566725 Fri, 16 May 2025 05:18:00 -0000 https://status.vanillo.tv/incident/566725#f94e48822ec1fb4ea0d01263d871d0a2a5a892a7be6637f8b21de6177c3e60df Due to an unforseen issue in the cluster maintenence, we are going to be placing the site into maintenance mode indefinitely to work out a fix. https://status.vanillo.tv/incident/566725 Fri, 16 May 2025 05:18:00 -0000 https://status.vanillo.tv/incident/566725#f94e48822ec1fb4ea0d01263d871d0a2a5a892a7be6637f8b21de6177c3e60df Due to an unforseen issue in the cluster maintenence, we are going to be placing the site into maintenance mode indefinitely to work out a fix. https://status.vanillo.tv/incident/566725 Fri, 16 May 2025 05:18:00 -0000 https://status.vanillo.tv/incident/566725#f94e48822ec1fb4ea0d01263d871d0a2a5a892a7be6637f8b21de6177c3e60df Due to an unforseen issue in the cluster maintenence, we are going to be placing the site into maintenance mode indefinitely to work out a fix. https://status.vanillo.tv/incident/566725 Fri, 16 May 2025 01:01:00 -0000 https://status.vanillo.tv/incident/566725#10562f775c636430009a04009157106b5bcf8cea9fadc2405a271fb8dde5458a We are performing maintenance on our cabernets clusters, which may lead to some down time. We apologize for any inconvenience, and welcome you to report any issues through our discord here: https://discord.gg/Xn7qv3MY https://status.vanillo.tv/incident/566725 Fri, 16 May 2025 01:01:00 -0000 https://status.vanillo.tv/incident/566725#10562f775c636430009a04009157106b5bcf8cea9fadc2405a271fb8dde5458a We are performing maintenance on our cabernets clusters, which may lead to some down time. We apologize for any inconvenience, and welcome you to report any issues through our discord here: https://discord.gg/Xn7qv3MY https://status.vanillo.tv/incident/566725 Fri, 16 May 2025 01:01:00 -0000 https://status.vanillo.tv/incident/566725#10562f775c636430009a04009157106b5bcf8cea9fadc2405a271fb8dde5458a We are performing maintenance on our cabernets clusters, which may lead to some down time. We apologize for any inconvenience, and welcome you to report any issues through our discord here: https://discord.gg/Xn7qv3MY https://status.vanillo.tv/incident/362555 Tue, 30 Apr 2024 01:32:00 -0000 https://status.vanillo.tv/incident/362555#348e83b2782129a064b2ae672e8e7c108f1145b329e4595258d9ebe09386be3a Services have been restored https://status.vanillo.tv/incident/362555 Tue, 30 Apr 2024 01:32:00 -0000 https://status.vanillo.tv/incident/362555#348e83b2782129a064b2ae672e8e7c108f1145b329e4595258d9ebe09386be3a Services have been restored https://status.vanillo.tv/incident/362555 Tue, 30 Apr 2024 01:16:00 -0000 https://status.vanillo.tv/incident/362555#0cf4bdac6a89e8ffd9e75db7938a04a862d61302b6f9dabb1585b7d2da487e05 We are currently investigating a disruption to the Vanillo.tv website and Api.vanillo.tv domain. We are working to resolve this as soon as possible, and will post updates when available https://status.vanillo.tv/incident/362555 Tue, 30 Apr 2024 01:16:00 -0000 https://status.vanillo.tv/incident/362555#0cf4bdac6a89e8ffd9e75db7938a04a862d61302b6f9dabb1585b7d2da487e05 We are currently investigating a disruption to the Vanillo.tv website and Api.vanillo.tv domain. We are working to resolve this as soon as possible, and will post updates when available https://status.vanillo.tv/incident/313165 Wed, 17 Jan 2024 05:20:00 -0000 https://status.vanillo.tv/incident/313165#329f9d070730c45e2c8bbdfb8912e15ff79f318ab9e4f4482807a61b74e18554 The incident has been resolved. Due to unforeseen issues with system configuration, restoration took longer than expected. https://status.vanillo.tv/incident/313165 Mon, 15 Jan 2024 21:30:00 -0000 https://status.vanillo.tv/incident/313165#2be4415ba520c5dd22a0febf22b519e7088dc8735aca4abe00dce41d4637d0d2 Power has been restored. We are working to bring our systems back online. https://status.vanillo.tv/incident/313165 Mon, 15 Jan 2024 17:40:00 -0000 https://status.vanillo.tv/incident/313165#7c573b5443d0c10d51370cd9c9e10d06c0ff2c4539411f03576e63415ac51033 Due to power issues, our Miami, Florida facility is without power. This has resulted in video uploads being temporarily paused from processing until resolution. https://status.vanillo.tv/incident/300231 Thu, 14 Dec 2023 01:53:00 -0000 https://status.vanillo.tv/incident/300231#54a4801ab95a739b76dc64adbbfc84b1bf6a9d062ab07bbf6c13b12007794443 We are back online now ==== We are experiencing an outage from Hetzner at the data center level We have engaged Hetzner support and are waiting for services to restore https://status.vanillo.tv/incident/300231 Thu, 14 Dec 2023 01:53:00 -0000 https://status.vanillo.tv/incident/300231#54a4801ab95a739b76dc64adbbfc84b1bf6a9d062ab07bbf6c13b12007794443 We are back online now ==== We are experiencing an outage from Hetzner at the data center level We have engaged Hetzner support and are waiting for services to restore https://status.vanillo.tv/incident/300231 Thu, 14 Dec 2023 01:53:00 -0000 https://status.vanillo.tv/incident/300231#54a4801ab95a739b76dc64adbbfc84b1bf6a9d062ab07bbf6c13b12007794443 We are back online now ==== We are experiencing an outage from Hetzner at the data center level We have engaged Hetzner support and are waiting for services to restore https://status.vanillo.tv/incident/291342 Thu, 23 Nov 2023 19:55:00 -0000 https://status.vanillo.tv/incident/291342#aa43e12b1edf2b2bf1f13628065ab78ab8f097d0b9c8b446e2b070ca4cb67041 Encoding Services Are Restored https://status.vanillo.tv/incident/288073 Fri, 17 Nov 2023 16:54:00 -0000 https://status.vanillo.tv/incident/288073#569f32edf58835f188ca79b60b3fbdb9a5c9f3a55e8afc989bca5d452acc5abb Video encodes are now available once again. https://status.vanillo.tv/incident/288073 Fri, 17 Nov 2023 15:45:00 -0000 https://status.vanillo.tv/incident/288073#24aec4cdc99a67a7cb4da36f191a2a3334ebee8ff2f510ee403eae3de2d2e2ae Power has been restored and we are working on recovering encoding services. https://status.vanillo.tv/incident/288073 Thu, 16 Nov 2023 13:20:00 -0000 https://status.vanillo.tv/incident/288073#ee0f4ef3d875dea4fc67eeabe6989c43d8585892331a08caa727263ecbf69d69 Widespread power outages are causing downtime at our servers located in Miami, Florida. Power delivery equipment has also been affected. Video encoding will be unavailable during this period. Video delivery and all other functions of the site will continue to function normally. https://status.vanillo.tv/incident/287272 Wed, 15 Nov 2023 01:02:00 -0000 https://status.vanillo.tv/incident/287272#244c12bd4fed9ad0e05116c0bd89b2451910ac1cd6fb3b73a076e86a991f8e37 We needed to make a change to the database which requires us to take the site offline for a short while Services will be restored as soon as possible https://status.vanillo.tv/incident/280022 Mon, 30 Oct 2023 20:11:00 -0000 https://status.vanillo.tv/incident/280022#80063d8527f705266c9cacedb264bd1242c227e11c1959d0d55fc9cc372fa1ac Cloudflare Workers outage causing downtime https://www.cloudflarestatus.com/incidents/l6x2h1zp69bc https://status.vanillo.tv/incident/272460 Fri, 13 Oct 2023 21:48:00 -0000 https://status.vanillo.tv/incident/272460#b241d9f165f622c7ffa08623ee2e418bfb9f5193b5abc94b08e44e12bdd132b0 Connectivity issues - resolved https://status.vanillo.tv/incident/272460 Fri, 13 Oct 2023 21:05:00 -0000 https://status.vanillo.tv/incident/272460#7841a1da7698c1f65c6928455c8de93dbd7da4394eb23906504933f1374c1610 We've deployed a new networking layer on our cluster and we're monitoring the results. We believe the issue to be resolved https://status.vanillo.tv/incident/272460 Fri, 13 Oct 2023 20:24:00 -0000 https://status.vanillo.tv/incident/272460#78ab4a4656a6cb4880fd71e4ffd4f43745d7978b4a91e97f9495ecc96718e21a Our cluster is experiencing networking issues between internal nodes and the outside world, due to a fault in an external provider. We're working to resolve the issue https://status.vanillo.tv/incident/269231 Sat, 07 Oct 2023 00:12:00 -0000 https://status.vanillo.tv/incident/269231#fcb22ae4941704b6c34c93df9f5dc314a8b3f9b247bd54c182325d6260d18f13 The issue has been resolved. Thank you for your patience. https://status.vanillo.tv/incident/269231 Sat, 07 Oct 2023 00:12:00 -0000 https://status.vanillo.tv/incident/269231#fcb22ae4941704b6c34c93df9f5dc314a8b3f9b247bd54c182325d6260d18f13 The issue has been resolved. Thank you for your patience. https://status.vanillo.tv/incident/269231 Fri, 06 Oct 2023 18:06:00 -0000 https://status.vanillo.tv/incident/269231#56932d1a3e8ba67d563bafb952a504c32376f2230dbf503ad61f5edacc2a7ac2 UPDATE: We have recovered all of the data for our database successfully. This means that no user data was lost in this process and Vanillo will be up as normal once we fully recover our Kubernetes clusters and bring them back to full operation. Thanks for your patience! https://status.vanillo.tv/incident/269231 Fri, 06 Oct 2023 18:06:00 -0000 https://status.vanillo.tv/incident/269231#56932d1a3e8ba67d563bafb952a504c32376f2230dbf503ad61f5edacc2a7ac2 UPDATE: We have recovered all of the data for our database successfully. This means that no user data was lost in this process and Vanillo will be up as normal once we fully recover our Kubernetes clusters and bring them back to full operation. Thanks for your patience! https://status.vanillo.tv/incident/269231 Fri, 06 Oct 2023 17:16:00 -0000 https://status.vanillo.tv/incident/269231#aeabac5452bc8d912b3a73177ab597686c50e38d1436251d08740ba07d649dcc Sometime within the last 24 hours, our database clusters suffered catastrophic failure due to a currently unknown reason. Per our preliminary investigation, it appears to be a bug with our Kubernetes operator, CloudNativePG, for our PostgreSQL database. This bug has caused all of our database clusters to stop working entirely, with no currently identified way to recover from the error or get the database files back. We are currently trying to find ways to resolve the scenario. The only usable backup currently is nineteen days old, as our automated backup system also appears to have failed around that point due to this bug. For now, the website will remain down while we work out what our next step shall be. We apologize profusely for the downtime, but unfortunately, this element was beyond our control. We are going to work as fast as we possibly can to resolve this and hopefully find a way to recover our most recent data rather than being forced to use an old backup. https://status.vanillo.tv/incident/269231 Fri, 06 Oct 2023 17:16:00 -0000 https://status.vanillo.tv/incident/269231#aeabac5452bc8d912b3a73177ab597686c50e38d1436251d08740ba07d649dcc Sometime within the last 24 hours, our database clusters suffered catastrophic failure due to a currently unknown reason. Per our preliminary investigation, it appears to be a bug with our Kubernetes operator, CloudNativePG, for our PostgreSQL database. This bug has caused all of our database clusters to stop working entirely, with no currently identified way to recover from the error or get the database files back. We are currently trying to find ways to resolve the scenario. The only usable backup currently is nineteen days old, as our automated backup system also appears to have failed around that point due to this bug. For now, the website will remain down while we work out what our next step shall be. We apologize profusely for the downtime, but unfortunately, this element was beyond our control. We are going to work as fast as we possibly can to resolve this and hopefully find a way to recover our most recent data rather than being forced to use an old backup. https://status.vanillo.tv/incident/223179 Thu, 22 Jun 2023 01:14:00 -0000 https://status.vanillo.tv/incident/223179#29d19f1fbb38359085cf0cc1a2bb4454d0b2d197de6de1680314a2e44c7078d9 The issue has been resolved. Sorry for the inconvenience. https://status.vanillo.tv/incident/223179 Wed, 21 Jun 2023 23:03:00 -0000 https://status.vanillo.tv/incident/223179#b17e406f96c1cbbb9f1a876f87fd17510685721b555244696a0737eb56be698e Our upstream storage provider is currently experiencing a service outage. They are actively working on resolving the issue. However, we do not have an estimated time for when the situation will be resolved. Thank you for your patience and understanding.