What is the maximum number of calendar events that Subscribe can handle? I have one calendar that generates 1003 appointments and it works, but the one with 1200 appointments fails. I also have another with around 3300 appointments and it fails too.

So, is 1000 events, give or take, the limit? Or is this machine dependent (ie: more memory, more events)?

Thanks!

-Rob

Hi Rob,

Thanks so much for reaching out to us here. While it's possible that the issue with your calendars could be related to the number of records, I'm not sure that is the case. How/when are the two calendars failing? Are you able to successfully "Test the Setup" in the Setup file, but are unable to subscribe to the calendars? Or does the test the setup not return event results?

Try duplicating one of your failing calendars, and add a filter to this new calendar, so that only a subset of the appointments are returned. Are you able to successfully subscribe to this filtered calendar? For more info on filters in Subscribe, see this link: http://www.seedcode.com/pmwiki/index.ph ... erCriteria

Thanks again,

Hey Dan, thanks for the reply. A bit more info. I'm not at the office now, so I can't test anything, but here's what prompted me to post. Our calendars are all pulled from the same Events table. Each event has a category and a completion status (it's basically a big to-do list). At first I was having trouble with categories not filtering. Punch Items are category P and Purchase Orders are PO. Filtering for P resulted in problems. As soon as I changed Punch to PU, everything worked. At that point I thought the problem was resolved.

Users are assigned to Projects and Projects have all kinds of sub-records like Punch Items, Tasks, Submittals, RFIs, Purchase Orders, Change Orders, Back Charges, etc. The idea was to generate four sets of calendars. Two sets would be user specific and the other two would reflect all projects. A user who wanted a Punch Items calendar could choose from the following:

My Projects Punch - Incomplete
My Projects Punch - Complete
All Projects Punch - Incomplete
Al Projects Punch - Complete

I wrote a script that set up all of the calendars in Subscribe and started testing them again using the "Test the Setup" button in Subscribe. I never attempted to subscribe to any of the calendars that failed. To verify the calendars, I performed a search of my Events table matched to the filter criteria. Subscribe would tell me that 378 calendar events were found and my found set would match. At first I couldn't figure out why "Test the Setup" was failing, but I soon realized all of the failed calendars were well over 1000 records. I attempted to recreate a few of the failed calendars manually, just to see if that made a difference, but they failed too.

When I get back to the office I'm going to try to further filter the failing calendars, maybe breaking them apart by year, and see if that helps. I'm also going to try subscribing to them to see if it works despite what "Test the Setup" reports. Basically 32 different calendars were created (8 different Event types x 4) and only a few failed and the ones that did all had in excess of 1003 records, so it just seems like a record limit issue to me...

Thanks!

Quick follow up...

I attempted to subscribe to one of the problem calendars and it didn't work. I was able to subscribe, but even after waiting for more than 10 minutes, no events showed up in Apple Calendar.

I then added an additional filter criteria to filter for events in 2015 only and the problem calendar worked. It finds 550 records.

So...I think there's a record limit issue. Any thoughts on this? Thanks!

Hi Rob,

Thanks so much for the additional info. I still think it's likely that something else is the cause of the issue here, not a record limit. We've tested thousands of records before, and I just successfully tested the setup for a calendar with over 10,000 events.

Are you seeing any info or an error message when testing the setup fails? The error message should hopefully shed some more light on the issue. Also, I would do a search for any empty values in your DateStart field. If there is one record in the found set with any empty DateStart, it could cause issues when testing the setup.

Thanks again,

Thanks for the prompt follow up, Dan. Hmmm...okay, that pretty much puts the record limit issue to rest. At least it confirms that many more than 1000 records are supported. So...I still wonder if it might be hardware related. My test server is an old Mac Mini that I've upgraded. It's an Early 2009 model with a 2 GHz Core 2 Duo, 8 GB of RAM, and an SSD, FYI.

I had problems with empty date fields when I first started setting up Subscribe, so I added a simple calc field, ShowOnCalendar, to the table that returns a 1 if the date is not empty and valid, otherwise 0. There are definitely records in this table that don't have a date. Not every to-do is assigned a due date, or the due date might be assigned later. I filter the calendar for records where ShowOnCalendar = 1, so empties should be removed.

I've performed a Find for invalid date or time (?) on the table it finds no records, so I don't think that's the issue.

I've attached two error messages. The first is the 0 events "Test the Setup" dialogue box that appears. I also happened to notice that after I get this 0 events message, the "Validate server settings" field on the Server settings tab returns an error. After a few seconds, when I click the "Validate server settings" button, the error disappears and I see "SUCCESS!" again. However, if I perform "Test the setup" again, I get 0 events again, and the error re-appears under "Validate server settings".

The "Validate server settings" error seems to be a timeout. Do you think this is related to the server hardware or my network? Again, adding an additional filter criteria to reduce the number of records to around 1000 always seems to make the problem calendars work.

Hi Rob,

Thank you for sending those screenshots -- interesting. It does appear to be a timeout issue, although I'm not sure if that would be due to your network or hardware.

Since smaller found sets seem to work without issue, try adding another rule to your "ShowOnCalendar" field to further limit the found set by date. For example, you might only need to show events that are dated one month ago or later. If that works for you / your users, this would likely limit the found set and let you publish your calendars as expected, like your other "smaller" calendars.

If that doesn't work for you, I'd be happy to take a closer look at a copy of your files, to try to replicate the issue on my machine. Feel free to get in touch with me directly at [email protected] if you'd like me to take a look.

All the best,

Thanks Dan. I'm going to test this on our live server later today and see if I still get the same problem. Is it possible to adjust the time in the Subscribe Engine, so that it takes longer to time out?

Sounds good, Rob. Keep me posted on what you find. It's not possible to adjust the timeout in the Subscribe engine, although I hope I can be more helpful after you get a chance to test this later today.

Thank you,

Sigh...doesn't work on the live server either. Smaller record sets work fine, but the larger calendars still report 0 records. After dismissing the 0 records dialogue, if I refresh either the field mapping or filters, I get the time out error. If I wait a few seconds and refresh again, it goes away.

At this point I need to decide what to do. Maybe this isn't the right solution for us? I love the simplicity, but I can't have calendars failing when an arbitrary number of records is reached. I can try to further limit the number of records with filters, but that's highly undesirable for a number of reasons. We already have a lot of calendars planned, so the idea of further breaking them down by year or something like that just creates complexity and confusion for our users...and for me, since I have to script all of it.

What would cause Subscribe to report 0 records? We've already addressed the possibility of an invalid timestamp by including a calculated field that determines whether the record is valid (1) or not (0). The only thing I've been able to observe consistently is the size of the filtered record set. All calendars with a set of records <1000 seem to work, while ones over 1000 seem to fail.

I decided to do a little further testing. I added a new table to my solution. There's a date field, calculated start and end timestamp fields (using the date field), a title field, and a flag that determines whether or not the record is valid. I added 25,000 records to the table. Subscribe failed. I then started to limit the number of valid records (anything before a date was valid, anything after was not) until I was able to get Subscribe to work. I determined that 13,499 records was the maximum number Subscribe would handle.

So there's obviously an issue with my existing Events table that is limiting the calendar to around 1000 events. Any thoughts on what that might be?

Hi Rob,

Thanks so much for the additional info. We are hitting a limit here, but it (clearly) isn't a limit on a number of records. It may vary with the server, and it will certainly vary depending on the payload in your events (for example, the length of the event descriptions).

Since there are no other options to increase that limit, you'll need to reduce the "payload" in your events (trim the description, for example) or reduce the found set. One option to reduce the found set would be NOT show events that are a certain number of days past the current date. (This assumes you may not need to see old events in your subscribed calendar.) That way, you should not need to setup up multiple calendars to represent one calendar, such as 1 calendar for each year as you suggested.

Please let me know if this helps, or if you'd like to strategize on how best to get these large calendars working.

Thanks again,

Hi again, Rob!

I also reached out to Todd Geist on this for other wisdom, and he offered this suggestion. It seems there is a way to increase your timeout! This involves a modification of the index.php file within the seedcode_subscribe_engine folder, so I would make a backup of that file before modifying it.

Let me know if this helps!

From Todd:

There are two things that could cause a time out. One is the actual time out setting and the other is the allowed memory. Both of these can be set on the server, and or overriden at the index.php level

Place this in index.php

----
ini_set('memory_limit', '128M');
set_time_limit ( 10 );
----

This will set the memory limit to 128Mb and the timeout to 10 seconds. You could set it to higher. Probably the upper limit is 512Mb, but it depends on the server's resources.

Thanks for additional information Dan. This experience has convinced me that I need to redesign the way I'm handling dates. My Dates table started simply enough but as we added more features and requirements changed, it became something of a mess. A few additional questions...

Do you suggest keeping the dates table used by Subscibe as simple as possible?

Do calculated fields in the table affect performance?

Does Subscribe work best when filtering is done on indexed/stored fields or can calculated fields be used without affecting performance?

What about fields mapped for the calendar? Is it better to use an indexed/stored field to display the event title and description or can calculated fields be used without affecting performance?

Basically what I'm considering doing is redesigning so the dates table is very simple and then storing the event's completion info, user assignment, date change history, etc. in related record(s).

Thanks again for all the help!

Hi Rob,

Glad to hear your follow up on this. I'm not sure if you are currently using many unstored calcs, or if you are considering moving to many calculations, but hopefully these answers help. Keep me posted with any other questions.

> Do you suggest keeping the dates table used by Subscribe as simple as possible?

Subscribe will be most impacted by the data "package" for the events (the fields that you map for the calendar subscription), not necessarily how many fields are in your dates table. However, if you have a lot of fields on the *layout* that you use for mapping your calendars (particularly if there are many unstored calcs -- see below), then your server might not be as efficient in gathering event data for the calendar subscriptions. I would suggest using a unique layout just for Subscribe calendars, with only the fields you need.

> Do calculated fields in the table affect performance?

Unstored calculations will almost certainly affect performance, but using them may sometimes be the only way to accomplish different filtering for some filtered calendars.

> Does Subscribe work best when filtering is done on indexed/stored fields or can calculated fields be used without affecting performance?

Since Subscribes filtering is essentially a FileMaker find, performance can be affected by using unstored calculations. But again, some filtering might only be accomplished by using unstored calculations. For example, to only show events from the last month forward, you could an unstored calculation that results to 1 *if* EventDate > Get(CurrentDate) - 30. Then, you could filter based on this calculation, only finding events when this calculation equals 1.

> What about fields mapped for the calendar? Is it better to use an indexed/stored field to display the event title and description or can calculated fields be used without affecting performance?

Indexed/stored fields would be best for this, although unstored calculations could also be used. I'm not sure what the performance impact would be in using unstored calcs for these mapped fields, but again, there most certainly would be some impact. At this point, I *think* the bigger issue for your calendars is the "payload"

> Basically what I'm considering doing is redesigning so the dates table is very simple and then storing the event's completion info, user assignment, date change history, etc. in related record(s).